Citrix NetScaler

How to monitor realtime traffic stats accurately on a Citrix NetScaler

on

The GUI on a Citrix Netscaler is nice but it takes 7 seconds to update and sometimes you need live stats and a 7 second delay is not going to work for you. You can do A LOT through command line you can’t through the GUI (isn’t that the case with pretty much anything though? :)). I am writing this article to show you how to get stats on each of your VIPS and servers quickly and also monitor live connections and traffic into your appliance.

 

 

GETTING DETAILS
First, you might want to get some details on your Load Balanced VIP. I am calling my Load Balanced VIP “lb_vsvr_name” in this example. Here is how you do that:

1. SSH into the Netscaler using Putty
2. Type:

sh lb vserver lb_vsvr_name

You will immediately see a lot of stats on how this VIP is configured like it’s IP, it’s status, connection method, persistency, bound service groups, etc. Pay attention to the bound service groups. So to get details on one of those, type this where “svcgrp_name” is the name of your service group:

3. Now type:

sh servicegroup svcgrp_name

This will show you details on the service group such as the status and the individual back end servers. Let’s say you have two servers called “svr_name1” and “svr_name2”. To get the details on those backend servers:

4. Type:

sh server svr_name1

and

sh server svr_name2

This will pull up stats on each of your backend servers.

 

GETTING REALTIME GENERAL PERFORMANCE STATS
1. Make sure you are still SSHed into the Netscaler using Putty
2. Type “shell” to enter the FreeBSD shell
3. Capitalization matters below or you will get an error. Type:

nsconmsg -s ConLB=2 -d oldconmsg

What this does is pull up realtime general stats on all your VIPs and services and continues to update it every 7 seconds until your press Ctrl+C to stop it. I know the display has a ton of info in it but Citrix actually has some very good documentation on how to read it all here: http://support.citrix.com/article/CTX113341

 

GETTING REALTIME TRAFFIC & CONNECTION STATS
Now that you have verified the VIP is up and have it’s IP and bound service groups. Here is how you can SSH into the Netscaler and get live traffic and connection stats for the VIP itself:

1. Make sure you are still SSHed into the Netscaler using Putty
2. Make sure you are not in the shell. Type “exit” if you are still in it from doing the previous section.
3. Now type:

shconnectiontable

This will give you every connection coming into the Netscaler. The columns displayed are in this order:

SRCIP = source IP address
SRCPORT = source port
DSTIP = destination IP address
DSTPORT = destination port
SVCTYPE = service type like HTTP, SSL, etc.
IDLTIME = idle time of that connection in milliseconds
STATE = current state of the connection, you will usually see it either waiting or established

Pretty hard to look at if the device you are looking at is in production and has a ton of traffic on it. So we need to filter this display. Let’s just look at how many connections we have to just one of the backend servers above. Type:

sh connectiontable | grep xxx.xxx.xxx.xx

or

sh connectiontable "ip == xxx.xxx.xxx.xx"

where xxx.xxx.xxx.xxx is the IP address of one of the backend servers above. This will give you stats on all the connections to that backend server.

1. Anything that says “MONITOR” is the Netscaler itself testing for uptime so ignore those.
2. If you see “MONITORS”, ignore those too, they are just SSL based monitors.
3. If you see “HTTP” or “SSL”, that’s the real traffic you are after. Note if they say “TIME_WAIT” or “ESTABLISHED”. If it’s live traffic, the state should be an established connection.

Let’s get into some more advanced filtering. Let’s say you want to filter all monitors to that IP and only display SSL, HTTP, etc:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR"

Let’s say you want to see all traffic to that IP but only established connections. Here is the syntax for that:

sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established"

Let’s say I want all the traffic to that IP, established connections only, and only display SSL connections:

sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established && svctype == SSL"

 

One thing to note, typically you can filter additional service types like this:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR && svctype != SSL"

but I found that if you have “MONITORS” and you wanted to filter them, it does not work and gives you an error saying it’s an invalid command. I actually have an open ticket with Citrix about this bug so it should be fixed in a future release. In the meantime, just use a combination of a filter expression and grep to filter it out. Here’s an example:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR" | grep -v MONITORS

 

FURTHER READING
If you need further info on commands and syntax available on the Netscaler, I have hosted the Citrix Netscaler Command Reference Guide for 9.1 here:

NS-CommandReference-Guide.pdf

Yes it’s 1742 pages but take the time to flip through it. It’s extremely powerful. The connectiontable related syntax begins on page 883 (873 in the guide). 🙂

About Jason Samuel

Jason Samuel lives in Houston, TX with a primary focus on strategic advisory and architecture of end-user computing, security, enterprise mobility, virtualization, and cloud technologies from Citrix, Microsoft, & VMware. He also has an extensive background in web architecture and networking over his 20+ year career in IT. He is an Author, Speaker, and Local User Group Community Leader. He is certified in several technologies and is 1 of 63 people globally that is a recipient of the prestigious Citrix Technology Professional (CTP) award. He is 1 of 42 people in the world that has been awarded as a VMware EUC Champion and VMware vExpert. He is a featured author on DABCC which provides the latest IT Community News on Cloud, Data Center, Desktop, Mobility, Security, Storage, & Virtualization. In his spare time Jason enjoys writing how-to articles and evangelizing the technologies he works with. Disclaimer: The content and opinions expressed in articles and posts are his own and are by no means associated with his employer.

Recommended for you

6 Comments

  1. Benni

    September 6, 2010 at 1:16 AM

    Very Nice Netscaler Infos.
    Thanks a lot.

  2. Pankaj Kumar

    July 24, 2013 at 12:29 AM

    Very informative …good job!

  3. Jason

    October 23, 2014 at 4:07 PM

    I would add the stat commands are also very useful if not critical for getting realtime performance info. For example:

    stat servicegroup

    Will show the stats for the individual servers in the service group including req/s and total numbers of connections and TTFB.

    I would also note for some reason in GUI 10.5 these numbers can seemingly no longer be found in the dashboard so getting them from the CLI is huge.

    Thanks,
    Jason

  4. Mario

    April 15, 2017 at 2:33 AM

    Thanks for sharing Jason! Keep it coming!

  5. shruti

    April 21, 2019 at 6:51 AM

    Is it possible to to monitor the netscaler service and service group in build 11.1 59.10

  6. shruti

    April 21, 2019 at 6:52 AM

    Like setting up alert notification or something like that

Leave a Reply

Your email address will not be published.