Citrix NetScaler

Best practices for Citrix Netscaler AAA logging and retention

on

41Z+JLHr+1L

By default the Netscaler is set to certain log levels for certain modules on the device, including AAA (authentication, authorization and accounting) logging. Sometimes you may want to change the AAA log retention temporarily for easier troubleshooting.

I’ve posted several articles around Netscaler AAA already but if you’re new to it, AAA logging is saved to the /var/log/ns.log file (and all subsequent rollover files i.e ns.log.0.gz, ns.log.1.gz, ns.log.2.gz, etc.) and you can view these via GUI by going to System > Auditing > then clicking on “Syslog messages”. Once the window loads, in the Module drop down choose “AAA” or whatever you are trying to filter for and hit “Find Now”. Choose the logs in the bottom left (just doubleclick).

ns0

The default setting for AAA logs is set to save the last 25 log files (circular logging where it will overwrite the oldest) and the size is set to 100 Kilobytes. Not a whole lot. The best practice is not to utilize the device itself for historical logs. It’s really only meant to be used for realtime/neartime logs for troubleshooting purposes. Historical Netscaler AAA logs should be offloaded from the appliance using a syslog server. It’s very easy to configure. Just go to Auditing > Syslog and add your syslog server and policy. Make sure you select “ALL” events. It’s always best to have more and let your syslog server do it’s thing than try and filter on the device and end up with something you needed missing. Forward it all. Your syslog server should be sized to take it and retain the data for whatever period your information security/compliance policy dictates.

If you don’t have a syslog server available, there’s no excuse. Stand up a free edition of Solarwinds Kiwi or Splunk: http://www.solarwinds.com/products/freetools/free-kiwi-syslog-server.aspx or http://www.splunk.com/view/SP-CAAAE8W. If you have multiple Netscalers you might already be running Citrix Command Center (which is also free) and has syslog ability: http://download.citrix.com/downloads/command-center.html.

Now, to actually view the settings for each log, WinSCP in and go to:

/etc

and copy over the newsyslog.conf. If you open it in Notepad it’s kind of hard to make out. Only use WinSCP to make a backup. Don’t try and edit it with Notepad and push it back.

To view and edit this file easily, use Putty instead and SSH in. Then go to:

1. shell
2. cd /etc
3. cat newsyslog.conf

ns1

Right along the top, there is a little warning. Any changes to this file are lost after rebooting. Whatever changes you make, you need to copy into the /nsconfig folder. By default there isn’t one in that folder until you copy it over.

This will show you the contents of the configuration file. See the line that says “/var/log/ns.log“? Those numbers to right are the settings for the ns.log. As I mentioned before, by default is set to save the last 25 log files and the size is set 100 Kilobytes. That’s not a whole lot.

4. Now before you decide to increase the size or number of logs files that are kept on the appliance, check to see how much disk space you have available. Type

df -h

What you’re interested is in the /var partition which you can see on my test device has plenty of room left in my screenshot.

ns2

5. Now let’s edit the newsyslog.conf. I don’t want to go overboard here. In this example I’ve calculated that if I bump the log from 100 K to 3 MB I should be okay. That’s 3 MB x 25 logs = 75 MB. That’s a little bit of extra logging I need for troubleshooting users who can’t login with out having to pull logs from my syslog server. Again, don’t max it out because the appliance itself is not where you want to be storing logs. To edit, type:

vi newsyslog.conf

and arrow your mouse down to the 100 and type “i” to edit/insert. Now type 3000 to make the size 3 MB. Once done, press the ESC key and type:

:x!

which will save and exit. If you screwed up and want to exit without saving your changes, type:

:q!

ns3

6. After you’ve made your changes, you need to copy the file to /nsconfig so it persists after device reboots.

7. Finally, you may need to restart the syslog process for the changes to go into effect (or just reboot the device if you have a maintenance window). In my experience I usually don’t have to but it’s always best to test to see if your changes persist through a reboot. Type the following to find the syslog process ID:

cat /var/run/syslog.pid

then to kill the process type:

kill -9

and finally to restart the process:

/usr/sbin/syslogd -b 127.0.0.1 -n -v -v &

This is all FreeBSD so if you’re no familiar with the switches:

-b will bind the syslog to the localhost
-n run in the foreground (as in wait until the process completes)
-v -v (yes that’s two of them) maxes out the verbosity of the logged messages

You can see below in my screenshot my process ID changes after the process is restarted. This is a good way to verify the process actually restarted:

ns5

8. Go back to the GUI syslog and check the latest log (hopefully it has turned over by now). You’ll notice the first entry in the logs will begin say:

logfile turned over due to size>3000k

or whatever size you had specified.

ns4

About Jason Samuel

Jason Samuel is an Infrastructure Architect in Houston, TX with a primary focus on mobility, virtualization, and cloud technologies from Citrix, Microsoft, & VMware. He also has an extensive background in web architecture and information security. He is certified in several technologies and is 1 of 50 people globally that is a recipient of the prestigious Citrix Technology Professional (CTP) award. He is 1 of 28 people in the world that is an Atlantis Community Expert (ACE). He is a featured author on DABCC which provides the latest IT Community News on Cloud, Data Center, Desktop, Mobility, Security, Storage, & Virtualization. In his spare time Jason enjoys writing how-to articles and evangelizing the technologies he works with.

Recommended for you

3 Comments

  1. Techno

    September 9, 2014 at 6:53 AM

    Nice Article

  2. Dave

    November 4, 2014 at 12:17 PM

    Excellent write up. What do you use to analyze your Netscaler logs? I’m looking for something that can help with quickly showing logon & logoff or timeout events.

  3. Jason Samuel

    December 29, 2014 at 2:15 PM

    Dave,

    You can setup filters in Splunk or Citrix Command Center. Filter for these event types:

    LOGIN_FAILED
    LOGIN
    LOGOUT

    You can also watch apps and virtual desktops being launched (the ICA session) by filtering for the following event types:

    ICASTART
    ICAEND_CONNSTAT

Leave a Reply

Your email address will not be published. Required fields are marked *