A Necessary Guide to the Avaya traceSM Utility

Necessity is the mother of invention.

Case in point is the Avaya utility, traceSM. As hard as I’ve tried, I have yet to find a halfway decent manual that explains how to use what is perhaps the most important Session Manager debugging tool. I would love to be proved inept in my efforts, but I’ve searched through every Session Manager document I can find. I have also asked people who work with traceSM on a regular basis if they’ve found anything useful and I always come up empty.

Therefore, I am documenting what I know in the hope that the next person who endeavors a search will find this blog article and have something to work with.

These two articles may help you better understand Avaya SIP and subsequently, traceSM.

The Steps Involved in Booting an Avaya SIP Telephone

Understanding Avaya’s Personal Profile Manager (PPM)

traceSM is an interactive perl script that allows an administrator to capture, view, and save call processing activity on a Session Manager. While not as powerful or versatile as wireshark, traceSM is absolutely essential when it comes to working with Avaya SIP. First off, it allows you to view SIP messages even if they have been encrypted with TLS. You cannot do this with wireshark. Second, traceSM allows you to see Avaya specific data such as Personal Profile Manager (PPM) messages and Session Manager call flows.

To run traceSM you need to Telnet into an active Session Manager. Personally, I use the TuTTy program, but you can accomplish the same thing with vanilla PuTTY.


By default, only one traceSM can be running on a Session Manager at any given time.  To invoke multiple instances, it must be run with the -m option.  However, Avaya advises against running multiple instance of traceSM in a production environment as it may cause performance problems.

traceSM -m

traceSM should be accessible through the default path, but in case you ever need to find it, look under /opt/Avaya/contrib/bin.  Remember, Linux is case sensitive.  It’s traceSM and never tracesm or Tracesm.


Since you are in a Telnet session, forget about using your mouse. It’s keystrokes only, Baby.

<UP>,<DOWN> Select a SIP/SM packet. Or scroll a large SIP packet when displaying the details
<HOME> Go to the first packet
<END> Go to the last packet. If the cursor is in the last packet while capturing packets, the screen will update with new arriving packets
<PGUP>,<PGDN> Page Up and Page Down
<LEFT>,<RIGHT> Move between different columns (IPs) when they don’t fit in the screen
<ENTER> Display the SIP/SM details. The SIP URI is highlighted in red, the SIP fields in blue and the content (e.g: SDP, xml) in green.
q Quit
f Display the Filter window to view/change filters
w Write the displayed (filtered) packets to a new file
s Start or Stop the capture
c Clear the screen
a Switch between SM and SM-100 perspective
i Switch between displaying Names or IPs in the column headers
r Switch between displaying RTP simulation or not
d Switch between SIP calls and display mode

It’s important to know that traceSM is a real-time capture tool. You cannot start traceSM and expect to see SIP packets that were sent prior to launching the tool. The capture begins when the application begins.

Also, traceSM’s capture buffer is limited in size to 10,000 packets and once that limit is reached, it stops collecting and displaying packets.  You will need to clear the buffer to get it moving again.

The following is an example of a typical traceSM screen. Note the handy list of commands at the bottom.


Also, take note of the words SIP, PPM, and CallP. SIP and PPM are in green and CallP is in red. This indicates that traceSM is currently only capturing SIP and PPM messages. You can change which packet and message types are captured with the “s” command.

In the following example, only SIP packets will be captured.  To capture PPM or Call Processing messages,”arrow” to the appropriate box and press Enter.  OK will restart the capture with the new parameters.


To view the expanded contents of a packet, up or down arrow to the packet you wish to view and press Enter.

Some SIP messages are so short that they can be completely viewed on the screen while others require you to arrow down to see everything.  The three dots at the bottom center of a message window will tell you if there is more to be seen.

In this example, I’ve selected to view the contents of an INVITE message.  Notice how traceSM displays different aspects of the SIP message in different colors — the Request Line is red, headers are blue, and the SDP is green.



A busy system will cause traceSM to capture and display packets faster than you can comprehend what is going on.  This is where the filter command comes in handy.  To create a filter, press “f” and select your desired options.

You can filter on everything from IP address to particular header values to specific SIP URIs or numbers.  For example, to only see the packets to or from extension 1902, you would set the following filter:

-u 1902

It’s often useful to create filters that eliminate “unnecessary” information.  For example, you can filter out OPTION messages (-no), registrations (-nr), subscriptions (-ns), and Session Manager’s call routing logic (-na).

By default, filters are logically added as an AND.  Specify -or to create either/or filters.

Specific to Avaya is the AV-Global-Session-ID header.  Filtering on this header allows you to do cradle-to-grave call tracing since the SIP Call-ID will change as a session moves through B2BUA entities, but AV-Global-Session-ID will not.

Filters are cleared by entering “f” without any options.

The following screen shot shows you the many different filters that can be created.


By default, traceSM will display ingress and egress points by name.  Use the “i” command to switch to display by IP address.


Use the “d” command to switch between displaying everything in the capture buffer to just the SIP calls.  Use “d” to then toggle back to a full display of all captured packets.

In this example you will see three SIP calls.  Two were successfully completed and one was rejected.


Earlier versions of traceSM did not allow you to capture and display PPM messages, but thankfully that has been resolved.  You need to enable their capture through the “s” command and traceSM will include them in the display.  As with SIP messages, pressing Enter  on a specific entry shows you the entire PPM message.


You can also trace the actions of Session Manager itself by capturing the Call-P messages.  Seeing Call-P messages can be valuable in debugging routing issues.


Saving Your Work

You can save the contents of the display buffer with the “w” command and clear the display buffer with the “c” command.  Saved traces can be later accessed with traceSM or exported to view with wireshark.

Files are written to the “home directory” of the logged in user.  For example, if you logged as “cust,” the file will be found in the /home/cust directory.

It is important to know that only the information currently shown on the traceSM screen is saved in the file.  This makes it easy to identify a problem with a filter, save the messages to a file, and then email the file to a support resource for further analysis.


If you find yourself in front of a Session Manager console screen and cannot recall all that I just tried to tell you, all you need to do is ask for help.

traceSM -h gives you the following information:


Mischief Managed

For some real-life examples of traceSM in action, please see these articles:

A traceSM View of Avaya Aura IMS Processing

Avaya Aura IMS Processing Once Again

A Look at how Avaya Aura Supports SIP-Based Voice Mail

A Look at SIP-Based Voice Mail Part Two

The best way to really learn a tool like traceSM is to use it and hopefully this brief guide will get you started.  Again, if anyone knows of a real document, please let me know.  Pretty much everything I’ve written here is what I’ve figured out on my own and the chances are high that I missed something.



  1. Marcelo · · Reply

    Hi Andrew,
    I have two questions: how to save my trace to re-open on wireshark? and It is possible to ‘upgrade’ the traceSM tool?

    Thanks for the guide, it is usefull.

  2. Marcelo, take a look at the “Saving Your Work” section and follow the steps to save your trace. The file will be in the form trace.tgz. A tgz file is a compressed TAR archive file. If you extract the files you will see a .pcapng file. Wireshare can open up and display files of that type.

    As for upgrade…the only way that I know to upgrade traceSM is to upgrade your Session Manager. However, I don’t know if Avaya has made any changes to the tool for a while.

    I hope this helps.

    1. Marcelo · · Reply

      Thank you for your answer. As you explain I usually save the trace with the ‘w’ command but it is in a text format, not .tgz even when I save as: file_name.tgz
      The version of SM is 6.2. Do you know if SM 6.3 is required?


      1. When you press “w” do you see this? Write the displayed (filtered) SIP packets to file.
        A .tgz file is created with a Text and Wireshark (.pcapng) version.
        File name (ENTER to cancel):

      2. Marcelo · ·

        Yes, but the file saved is not .tgz. Looking on my home directory there is no .tgz only the files saved that I can open in text format. Maybe is my SM version.
        Thank you Andrew

      3. Marcelo, I am not sure what your problem is. I just logged onto a SM, ran traceSM, captured some packets, and did a “w” to save them. It asked me for a file name (I chose “andrew”) and then I did a “q” to exit traceSM. I did an ls from the current directory and I see the file andrew.tgz.

  3. Marcelo · · Reply

    Andrew, don’t worry. Thank you for your time.

  4. Hi Andrew, every time we open the capture in Wireshark, it complains: the file “…”isn’t a capture file in a format Wireshark understands.
    Our Session Managers are on 6.3.8, our Wireshark is on latest version.

    Could you please check if this is a bug?

    1. Terry. Are you getting a tgz file (tar archive)? That’s what I get when I save a trace. Inside that file you will find a .pcapng file. Wireshark is able to read that file.

      Are you getting something different?

  5. Thanks for your reply Andrew. Yes, we got tgz file, and downloaded it via WinSCP.
    However Wireshark doesn’t like the .pcapng in it.
    We are doing SBC troubleshooting at the moment.
    Both myself and our maintenance provider / Avaya technician are having same issue.

    We have two session managers, both are on 6.3.8

    1. Terry, for what it’s worth, I use a program called 7-Zip File Manager to open the tgz file. From inside 7-Zip I open the embedded tar. I then double-click on the pcapng file to launch Wireshark. It always works for me.

    2. Johan Vandekerckhove · · Reply

      Hi Terry, I have the same issue, wireshark can’t handle the pcap file from TraceSM. I wonder what causes this.

  6. Hi Andrew, thanks for your reply. I tried WinRAR and 7-Zip, on Windows 7 and Windows 8.
    I will lodge a ticket with Session Manager support

    1. Terry, if you’d like to send me the file I can also take a look at it. My email address is on the main page of this blog.

      1. James Wilson · ·

        i am having the exact same issue. Is there a solution to this? im using 7-zip to open the .tgz

      2. I’m not sure why it isn’t working for you. I haven’t done it for a while, but it always worked from me. Are you on an old version of SM?

  7. Olivier Rime · · Reply

    Hello, what are the exact meanings of the T:, F:, U:, and P: fields in Avaya SIP traces? T: and F: are quite clearly “To” and “From”, but I wonder about U: and P:.

    14:58:43.419 |–INVITE–>| | | (106) T:003332244xxxx F:+3332235xxxx U:003332244xxxx P:terminating


    1. Olivier, here is a list of the recognized compact header forms for SIP: http://www.iana.org/assignments/sip-parameters/sip-parameters.xhtml.

      I don’t see enough your message to clearly understand what is happening in your trace. Is U: a header or a parameter?

  8. Olivier Rime · · Reply

    The F: T: U: and P: fields appear in the traceSM view on the single line corresponding to the INVITE message. Here the trace with the message detailed (you will want to copy paste in a text editor for better readability:

    EUACM(.10) EU-BtalkSBC1
    15:56:32.760 |–INVITE–>| | | (2) T:003332291xxxx F:+3332235xxxx U:003332291xxxx P:terminating
    15:56:32.762 || | (2) T:003332291xxxx F:+3332235xxxx U:003332291xxxx P:terminating
    15:5| –TLS-> |
    15:5|INVITE sip:003332291xxxx@euc.[customer].com SIP/2.0 |
    15:5|From: “DUPONT,N” ;tag=0e85495a948e4182fc545a6c200 |
    15:5|To: |
    15:5|Call-ID: 0e85495a948e4183fc545a6c200 |
    15:5|CSeq: 1 INVITE |
    15:5|Max-Forwards: 71 |
    15:5|Via: SIP/2.0/TLS;branch=z9hG4bK0e85495a948e4184fc545a6c200 |
    15:5|Via: SIP/2.0/TCP;branch=z9hG4bK0e85495a948e4184fc545a6c200 |
    15:5|Supported: 100rel,histinfo,join,replaces,sdp-anat,timer |
    15:5|User-Agent: Avaya CM/R016x. |
    15:5|Contact: “DUPONT,N” |
    15:5|Route: |
    15:5|Accept-Language: en |
    |Alert-Info: ;avaya-cm-alert-type=internal |
    |History-Info: ;index=1 |
    |History-Info: “003332291xxxx” ;index=1.1 |
    |Min-SE: 1200 |
    |P-Asserted-Identity: “DUPONT,N” |
    |Record-Route: |
    |Session-Expires: 1200;refresher=uac |
    |P-Charging-Vector: icid-value=”AAS:7391-9554e8001e448a9554fc81c2a6″ |
    |Av-Global-Session-ID: 00e85495-a948-4401-80fc-5405a6c20000 |
    |Content-Type: application/sdp |
    |Content-Length: 221 |
    | |
    |v=0 |
    |o=- 1411134992 1 IN IP4 |
    |s=- |
    |c=IN IP4 |
    |b=AS:64 |
    |t=0 0 |
    |a=avf:avc=n prio=n |
    |a=csup:avf-v0 |
    |m=audio 16392 RTP/AVP 8 101 |
    |a=rtpmap:8 PCMA/8000 |
    |a=rtpmap:101 telephone-event/8000 |
    |a=ptime:20 |

    1. Interesting. These aren’t SIP headers, but some form of Session Manager nomenclature. Unfortunately, I am not able to get to an Aura system today to dig deeper. I should have an opportunity to do that next week and if I can, I will. Until then, I am guessing that it’s internal SM call processing steps.

    2. I’ve been thinking about this and I expect that the P refers to the IMS Phase that SM goes through when processing a call — IMS-Orig, Orig-Done, IMS-Term, and Term-Done. I bet this is the termination (called party) phase of processing.

  9. Hi Andrew – love your blogs! Question – what sequence should I use to capture a trace on an extension? I have tried several and never see any data on the screen. Second question – how do I deleted written files – even if they are empty?

    1. Sandy, thanks for the compliment.

      You would need to create a filter using the -u option. See my explanation above. Make sure you filter on the right extension.

      To delete files, you need to go to the Linux prompt and remove the files from the traceSM directory.

      Does this help?

  10. Interesting blog..i have learn basis to put the SIP traces on the Session manager.

    thanks for sharing the knowledge.

  11. hi Andrew, nice guide. my question is, its posible change the buffer for a bigger number??? and how i can change?

    1. Sorry, but the buffer size is fixed.

  12. Johan Vandekerckhove · · Reply

    to unzip a tar.gz file, just do a ‘tar -xvf filename.tar.gz’ in the same folder. Then you’ll have the .pcap and traceSM file. The traceSM file can be re-opened offline with the command ‘traceSM filename’. Useful for troubleshooting!

  13. Hi Andrew, how can i decrypt trace downloaded? i have the trace but when i open in wireshark its encrypted with SSL, how can i decrypt it? …Best Regards.

  14. Hi Andrew, nice post…

    is there any documentation about step-by-step SM rules to route a sip-call from incomming origin sip-entity INVITE to outgoing INVITE to destination sip-entity…
    1st Orig Contact
    2nd Orig Location
    3rd Ingress Adaptations

    Thanks in advance!

  15. Thankyou Andrew, but I see nothing about SM routing criterion (internal SM call processing steps), just SM administration, I need something more specific. Any help, will be wellcome! Best regards

  16. Miguel ( MIke) · · Reply

    Hello Andrew, Is there a way to load the saved files on the traceSM back? the ones you save with W?


    1. If there is a way, it would be from the command line, but it’s not something I’ve ever tried.

    2. Johan Vandekerckhove · · Reply

      Andrew, see my comment above. You can open any saved logfile for offline troubleshooting

      1. Thanks Andrew for the post. I’m glad you like traceSM.
        Alternatively, you can also run traceSM in Windows under Cygwin, so you can open a previously capture .tgz file (“w” option) in a Windows PC as well.

      2. And traceSM can also be run from an Apple MAC. In both the Cygwin/MAC case, need to get the traceSM from the Session Manager server and copy it to your PC/MAC. The MAC support was added in traceSM v3.20 which is included in SM 6.3.18 and later.

  17. Hola Andre me puedes ayudar con el comando para matar la sesión de traceSM ya que cuando ejecuto el comando traceSM me dice:
    traceSM is already running. only one instance is allowed

    1. I don’t know a way to kill traceSM from a remote Putty instance. I would think it would just timeout on its own.

      1. Try traceSM -k

      2. Alejandro · ·

        “traceSM -k” should kill all existing sessions. Then run traceSM again as usual.

      3. Good to know! Thanks.

  18. Peter Kiss · · Reply

    Hi Andrew,
    You can try ” traceSM -max 100000 ” to increase buffer size.
    Just like multiple instance switch, it is not recommended in prod environment because of performance issues.

    1. hi peter its posible enter traceSM -max 20000?

  19. Thanks for your effort , did you have video for this.

    1. Thanks. No video for this. Lots for Breeze, though.

  20. Avaya user · · Reply

    I accidentally deleted tracer_asset_log files from /var/log/Avaya/trace directory. Now when I run traceSM it runs but does not capture or display anything.. capture 0 display 0 alway …can some one please help

    1. Alejandro · · Reply

      You need to restart the rsyslog service “service rsyslog restart” in order for that file to be recreated. It is not traceSM itself the issue, but the underline SM processes that writes that log uses rsyslog.

  21. hi i have a TraceSM is already running – how to stop it?

    1. It’s right there in the article.

  22. thaks i don´t saw it, but the command didn´t works traceSM -k devolve me:

    [root@SMan ~]# traceSM -k
    Unknown option: k
    traceSM V3.6

    and the other possible options can i works like -u -i -c….

    con you help me.

  23. Michael Waldon · · Reply

    Andrew, I have read both your posts on TraceSM and TraceSBC but I’m having difficulty in determining when you would use one vs. the other. Will you provide insight on this please?

    1. Depending on what you are trying to solve, they are both useful. You can look use TraceSBC on the ingress and egress of SIP messages into the Avaya system and you can use TraceSM as those messages move around Aura. TraceSM is probably more useful, but that may not always be the case.

  24. Lowell Miller · · Reply

    Andrew: Do you know why I would get this response when I try to go to Session Manager and run the Trace SM tool ?

    Connecting to XX.XX.XX.XX


    [SSH] Protocol Version 2 (OpenSSH_7.4)
    [SSH] no matching cipher found: client aes128-cbc,3des-cbc,blowfish-cbc,arcfour,cast128-cbc server aes128-ctr,aes192-ctr,aes256-ctr


    My connection is dropping immediately and at this point in time I am not sure why.
    I am able to connect to a different Session Manager jut not this one in particular.

    1. I don’t know. Is it possible to reboot the server?

  25. Hi Andrew,

    Thanks for sharing the excellent stuff on both the traceSBC & traceSM.

    I got an issue with a customer as the calls will disconnect if they connect from an Avaya IP phone to a Meetme conference bridge. This time the call was good enough for 45 minutes (no fix time of dropping the calls) and then dropped from their end. I believe it’s SIP issue. Please can you advise about the feasible solution.


    1. I would start by looking at call traces.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: