Getting around Netscaler Client IP issues without having to use USIP

July 16th, 2010 Jason Samuel 1 comment

If you work with Netscalers, you already know how SNIPs and MIPs work. They are the IPs that communicate with your servers. Every packet from the outside world that passes through the Netscaler will hit your server as though it was coming from this IP. This is the default behavior on a Netscaler.

Unfortunately if your servers require the client’s true IP address, all you will see for every client will be the SNIP or MIP address you are using. Here are a few examples I have personally had to deal with:

- IIS requires client IP for IIS logs
- IIS requires client IP for ACL filtering
- Web application requires client IP for it’s own logging purposes
- Web application requires cilent IP for authentication

Now to get around this for IIS, you can install ISAPI filters and set your Netscaler to use a custom header to store the true client IP address and pass that along with every packet. The ISAPI filter in IIS would look for this particular header and log that instead of the SNIP/MIP. There’s a very nice Client IP extraction ISAPI filter with some great instructions located on this page from Citrix or you can even write your own:

http://support.citrix.com/article/CTX119347

This is great for logging but sometimes your web application may grab the IP address from web server variables. To return the IP address of the host making the request, the server variable that your ASP, ASP.NET, and PHP code will call is “REMOTE_ADDR”. Unfortunately REMOTE_ADR is going to have your SNIP IP and not the client IP when passing through the Netscaler. The good news is that if you have control of your code, you can make it call from a different location that includes the client IP your ISAPI filter is passing. So if your code currently uses:

Request.ServerVariables(“REMOTE_ADDR”)

then switch it to:

Request.ServerVariables(“http_client_ip”)

where “http_client_ip” is the real client IP address and not the SNIP. Everything will work fine after that.

Here’s a nice snippet of code you can run on your web server and hit from your client browser to see what all the Netscaler is passing to the server:

<html>
<body>
<p>
NS SNIP or MIP IP address:
<%Response.Write(Request.ServerVariables(“remote_addr”))%>
<br>
<b>The NS Inserted Client IP:</b>
<%Response.Write(Request.ServerVariables(“http_Client_ip”))%>
<br>
Accept Encoding:
<%Response.Write(Request.ServerVariables(“HTTP_ACCEPT_ENCODING”))%>
<br>
Cookies:
<%Response.Write(Request.ServerVariables(“HTTP_COOKIE”))%>
<br>
</body>
</html>

(***Note, I apologize in advance if WordPress messes up the code block above. It usually changes the quotes in the code around so if you have issues with it, I will host a .txt file here with the code. Just comment to this post if you need it.)

This is all well and good when you have access to the code and can make this change but what about web apps that are 3rd party and you cannot modify the code? Well then you have no choice but to use the USIP (Use Source IP) feature on the Netscaler. What this does is pass the client IP straight through the Netscaler to your backend server so that the SNIP or MIP you are using on your Netscaler is never seen by the server (except for monitor probes from the Netscaler itself of course). This option is disabled by default. It can be applied to the entire Netscaler or turned on and off per service.

Best practice is not to use it and avoid it as much as possible. The reason is that using the USIP feature means you are going to lose very important features on the Netscaler such as connection multiplexing and surge protection. It’s always better to use the Client IP header insertion instead. In fact to date I have personally never enabled USIP in any environment I have built so far.

Here’s a very excellent and recent article from Citrix on the pros and cons of using USIP mode:

http://support.citrix.com/article/CTX121974

I hope this helps anyone trying to get around Client IP address related issues. Please feel free to post a comment on your own experiences or suggestions.

Categories: Apache, Citrix NetScaler, IIS Tags: , ,
| More

How to monitor realtime traffic stats accurately on a Citrix Netscaler

July 9th, 2010 Jason Samuel 1 comment

The GUI on a Citrix Netscaler is nice but it takes 7 seconds to update and sometimes you need live stats and a 7 second delay is not going to work for you. You can do A LOT through command line you can’t through the GUI (isn’t that the case with pretty much anything though? :) ). I am writing this article to show you how to get stats on each of your VIPS and servers quickly and also monitor live connections and traffic into your appliance.

 

 

GETTING DETAILS
First, you might want to get some details on your Load Balanced VIP. I am calling my Load Balanced VIP “lb_vsvr_name” in this example. Here is how you do that:

1. SSH into the Netscaler using Putty
2. Type:

sh lb vserver lb_vsvr_name

You will immediately see a lot of stats on how this VIP is configured like it’s IP, it’s status, connection method, persistency, bound service groups, etc. Pay attention to the bound service groups. So to get details on one of those, type this where “svcgrp_name” is the name of your service group:

3. Now type:

sh servicegroup svcgrp_name

This will show you details on the service group such as the status and the individual back end servers. Let’s say you have two servers called “svr_name1″ and “svr_name2″. To get the details on those backend servers:

4. Type:

sh server svr_name1

and

sh server svr_name2

This will pull up stats on each of your backend servers.

 

GETTING REALTIME GENERAL PERFORMANCE STATS
1. Make sure you are still SSHed into the Netscaler using Putty
2. Type “shell” to enter the FreeBSD shell
3. Capitalization matters below or you will get an error. Type:

nsconmsg -s ConLB=2 -d oldconmsg

What this does is pull up realtime general stats on all your VIPs and services and continues to update it every 7 seconds until your press Ctrl+C to stop it. I know the display has a ton of info in it but Citrix actually has some very good documentation on how to read it all here: http://support.citrix.com/article/CTX113341

 

GETTING REALTIME TRAFFIC & CONNECTION STATS
Now that you have verified the VIP is up and have it’s IP and bound service groups. Here is how you can SSH into the Netscaler and get live traffic and connection stats for the VIP itself:

1. Make sure you are still SSHed into the Netscaler using Putty
2. Make sure you are not in the shell. Type “exit” if you are still in it from doing the previous section.
3. Now type:

shconnectiontable

This will give you every connection coming into the Netscaler. The columns displayed are in this order:

SRCIP = source IP address
SRCPORT = source port
DSTIP = destination IP address
DSTPORT = destination port
SVCTYPE = service type like HTTP, SSL, etc.
IDLTIME = idle time of that connection in milliseconds
STATE = current state of the connection, you will usually see it either waiting or established

Pretty hard to look at if the device you are looking at is in production and has a ton of traffic on it. So we need to filter this display. Let’s just look at how many connections we have to just one of the backend servers above. Type:

sh connectiontable | grep xxx.xxx.xxx.xx

or

sh connectionable “ip == xxx.xxx.xxx.xx”

where xxx.xxx.xxx.xxx is the IP address of one of the backend servers above. This will give you stats on all the connections to that backend server.

1. Anything that says “MONITOR” is the Netscaler itself testing for uptime so ignore those.
2. If you see “MONITORS”, ignore those too, they are just SSL based monitors.
3. If you see “HTTP” or “SSL”, that’s the real traffic you are after. Note if they say “TIME_WAIT” or “ESTABLISHED”. If it’s live traffic, the state should be an established connection.

Let’s get into some more advanced filtering. Let’s say you want to filter all monitors to that IP and only display SSL, HTTP, etc:

sh connectiontable “ip == xxx.xxx.xxx.xxx && svctype != MONITOR”

Let’s say you want to see all traffic to that IP but only established connections. Here is the syntax for that:

sh connectiontable “ip == xxx.xxx.xxx.xxx && state == established”

Let’s say I want all the traffic to that IP, established connections only, and only display SSL connections:

sh connectiontable “ip == xxx.xxx.xxx.xxx && state == established && svctype == SSL”

 

One thing to note, typically you can filter additional service types like this:

sh connectiontable “ip == xxx.xxx.xxx.xxx && svctype != MONITOR && svctype != SSL”

but I found that if you have “MONITORS” and you wanted to filter them, it does not work and gives you an error saying it’s an invalid command. I actually have an open ticket with Citrix about this bug so it should be fixed in a future release. In the meantime, just use a combination of a filter expression and grep to filter it out. Here’s an example:

sh connectiontable “ip == xxx.xxx.xxx.xxx && svctype != MONITOR” | grep -v MONITORS

 

FURTHER READING
If you need further info on commands and syntax available on the Netscaler, I have hosted the Citrix Netscaler Command Reference Guide for 9.1 here:

NS-CommandReference-Guide.pdf

Yes it’s 1742 pages but take the time to flip through it. It’s extremely powerful. The connectiontable related syntax begins on page 883 (873 in the guide). :)

| More

Making a Logitech Harmony 900 remote IR mini blaster extension cable

July 3rd, 2010 Jason Samuel 2 comments

Though not IT related, this might help a fellow home theater aficionado. I use a Logitech Harmony 900 for my home theater room. This is Logitech’s latest RF remote which you can see here: http://www.logitech.com/en-us/remotes/universal_remotes/devices/5874

As pictured above, it comes with an IR receiver and two IR mini blasters which convert the RF signal to IR. These blasters are 6 ft. in length allowing you to stick them practically anywhere to control your AV equipment. My dilemma 6 ft. was not enough to run one of these mini blasters from the IR base receiver through the wall, through the ceiling, and down into the home theater room to control the projector.

Since these IR mini blasters have 2.5 mm jacks that connect them to the base RF receiver, I figured I could try using an extension cable and hopefully not lose any signal in the process. I bought a 12 ft. extension from Amazon for just $5.99. It was worth a shot:

http://www.amazon.com/gp/product/B000Z0YECU/ref=oss_product?tag=amaz98-20

I attached the extension cable to the mini blaster end and then the other end of the extension cable to the base receiver. That effectively gave me an 18 ft. mini blaster. Ran it through the wall and ceiling, then used double sided mirror tape to attach the blaster portion to the ceiling pointed at the projector which is a good 15 ft. to the back of the room. It works flawlessly and it’s barely noticeable attached to the ceiling! I hope this IR extension cable helps anyone that was in the same situation as me. Please do comment if it works for you. :)

Categories: Uncategorized Tags:
| More

MSMQ messages stuck in the outgoing queue after cloning servers

June 30th, 2010 Jason Samuel 3 comments

Always remember to sysprep when cloning a node! MSMQ (Microsoft Message Queuing) uses a registry valued called QMId located at:

HKLM\Software\Microsoft\MSMQ\Parameters\Machine Cache

which has to be unique on all your servers or it will cause all sorts of issues. Sysprepping a server generates a new SID which also generates a new QMId in the process. If you don’t do this and have multiple servers out there with the same QMId, you will see strange things like messages remaining in the outgoing queue in limbo and sometimes just disappear completely with no trace in journaling anywhere. Bottom line, don’t confuse MSMQ! It’s fragile!

Thankfully John Breakwell (MSMQ guru at Microsoft) blogged about this exact issue which made me realize what was going on in the environment I was troubleshooting. Well done John!

http://blogs.msdn.com/b/johnbreakwell/archive/2007/02/06/msmq-prefers-to-be-unique.aspx

I simply ran sysprep and resealed the clone to fix it. It automatically generated a new QMId in the process so I didn’t have to follow the exact steps John had blogged about. It was a simple mistake which can be avoided in the future by using the Guest Customization Wizard in VC to automatically sysprep when creating the clone.

BTW, you can check if your SIDs are identical on a server by using getsid.exe. Download the Windows XP Service Pack 2 Support Tools:

http://www.microsoft.com/downloads/details.aspx?FamilyId=49AE8576-9BB9-4126-9761-BA8011FABF38&displaylang=en

and use the getsid.exe tool to compare SIDs on your Windows servers. Usage syntax is here:

http://technet.microsoft.com/en-us/library/cc784314%28WS.10%29.aspx

Since I’m blogging about MSMQ, let me take a moment to plug QueueExplorer (http://www.cogin.com/mq). This is hands down the best MSMQ management software I have ever used and the developer is very open to feature requests. Give them some business and tell them Jason sent ya! :) Check out a screenshot of it below:

| More

How to REALLY set a Netscaler back to factory defaults (factory new condition)

June 29th, 2010 Jason Samuel 2 comments

This is my follow-up to my original article on setting a Netscaler back to factory defaults (http://www.jasonsamuel.com/2010/06/22/how-to-reset-a-citrix-netscaler-back-to-factory-defaults/). The instructions I provided wiped the config but what about if your actual Netscaler system is corrupted and you need to bring it back to factory new conditions? Well here is the procedure which is also outlined here on CTX116742 (http://support.citrix.com/article/CTX116742). ZebOS by the way if you are interested is what does the routing on the Netscaler (http://www.ipinfusion.com/products/products_home.html). This is a commercial version of GNU Zebra (http://www.zebra.org). Development on Zebra stopped back in 2005 and Quagga became the successor (http://www.quagga.net). A lot of network appliances out there are based on these two.

1. After completing the following steps, the NetScaler will have no configuration left and will be ready for the next installation or environment. Connect to the device using the serial console or with a Secure Shell client (SSH). Delete the system, routing, and SSL configuration:

> shell
# cd /etc
# for file in *; do
# rm /nsconfig/$file 2> /dev/null
# done
(Ignore warnings about files or directories missing. Not all files in /nsconfig are present in /etc)
# cd /nsconfig
# rm ns.conf* ZebOS.conf rc.conf
(Ignore warnings about files or directories missing)
# cd ssl
# GLOBIGNORE=”ns-*”
# rm *
# exit
> reboot

2. Perform an initial system configuration. Connect to the device using the serial console:

> set ns config –ipaddresss nsip -netmask netmask
> save config
> reboot

Categories: Citrix NetScaler Tags:
| More

Setting the date, time, and time zone in FreeBSD on your Citrix Netscaler

June 28th, 2010 Jason Samuel 5 comments

I recently troubleshot an issue where cookie persistency was not working properly when failing over from the active node to the passive node in a Citrix Netscaler HA cluster (high availability cluster). I found the issue was that if the date and time on your appliances are not the same, the cookie will be invalid causing persistency to fail when failing over to the passive node. That’s a big issue because all your users will lose their sessions during a failover.

To quickly check your appliances:

1. SSH using Putty into your NSIPs of your Netscalers.

2. Type “shell” to enter the FreeBSD shell

3. Type “date” to see the current date, time, and time zone. Both your Netscalers should be identical.

4. If they are not identical, set the time by typing the current date and time in the following format:

date yymmddhhmm.ss

For example, to set the date to June 28, 2010 12:13 PM with 15 seconds left in the minute:

date 1006281213.45

You will get this result if you are successful:

Mon Jun 28 12:13:45 CST 2010

5. Now if your timezone says GMT or something instead of your correct time zone, then you need to navigate to timezone file directories by typing:

cd /usr/share/zoneinfo

6. Type this to list all the different zone folders:

ls-l

7. To set the timezone, copy your timezone file to the localtime folder. Below, I am setting the timezone to CST (Central Standard Time):

cp /usr/share/zoneinfo/America/Chicago /etc/localtime

8. Now just type “date” again and it should be the correct time, date, and timezone. That’s it, you’re done! Let me know if you run into any trouble. :)

| More

How to reset a Citrix Netscaler back to factory defaults

June 22nd, 2010 Jason Samuel 2 comments

On older Netscaler appliances, the command to clear your Netscaler configuration and reset it to factory defaults used to be:

clear config –c y

But on newer devices, the command is:

clear ns config level

where level is one of the following:

1. basic – which clears everything except NSIP, MIPs, SNIPs, network settings, HA, features, and the nsroot account

2. extended – which clears everything except NSIP, MIPs, SNIPs, network settings, & HA

3. full – which clears everything except teh NSIP and default gateway

So typically, I type this to do a full wipe:

clear ns config full

You can also do this through the GUI by going to System > Diagnostics > Maintenance > Clear Configuration. Use the drop down box to select basic, extended, or full then click the Run button.

With that said, I have run across newer devices (even running NS 9.1 and newer) that just don’t clear all their config info. For those, I have just done the following to reset:

1. SSH using Putty into the Netscaler and login
2. Type “shell” to enter shell
3. cd /nsconfig
4. Rename the configuration by typing “mv ns.conf ns.conf.bak”
5. Type reboot

This should reset the device to the IP of 192.168.100.1 with the nsroot/nsroot account credentials. Your licenses and SSL certs will remain though since these are files and not part of the Netscaler config. You will have to delete them manually via WinSCP.

UPDATE 6/29/10: To completely reset your Netscaler to “factory new” conditions, it’s a little bit more work but I have outlined the procedure here:

http://www.jasonsamuel.com/2010/06/29/how-to-really-set-a-netscaler-back-to-factory-defaults-factory-new-condition/

Categories: Citrix NetScaler Tags: ,
| More

How to update your Citrix Netscaler firmware

June 22nd, 2010 Jason Samuel 2 comments

I’m doing some firmware updates on Citrix Netscaler appliances today and thought I would give you guys a quick guide on how to do this. This applies to all legacy appliances, MPX appliances, and VPX appliances. You use the same procedure for both Classic and nCore firmware packages. It is very easy to do. :)

1. Login to MyCitrix at http://citrix.com/English/mycitrix/index.asp and download the firmware .tgz package. Make sure to check compatability with your appliance before downloading.

2. Use WinSCP to transfer the package to the /var/nsinstall directory on your appliance.

3. SSH into your appliance using Putty and type “shell” to enter the shell

4. Type cd /var/nsinstall

5. Type ls to list the contents of the directory and verify your newly uploaded firmware package is there

6. Now type “tar -zxvf build-xxxxxxxx.tgz” to unzip the package

7. Then type “installns” to begin the install. Should take about 30 seconds to copy to flash and install.

8. Reboot the appliance once prompted for it (just type Y).

9. Login via the web GUI and verify the firmware version in the top right corner is the same as the package you were trying to apply.

Categories: Citrix NetScaler Tags: ,
| More

How to fix an IP address conflict on your network caused by a VM using PowerCLI

June 11th, 2010 Jason Samuel No comments

IP Address conflict? Have you narrowed the MAC down to being a VM in one of your ESX/vSphere clusters? Well use vSphere PowerCLI to figure out what device and adapter is causing the conflict.

Example, if your MAC is 005056b60b13:

Get-vm | Select Name, @{N=“Network“;E={$_ | Get-networkAdapter | ? {$_.macaddress -eq “00:50:56:b6:0b:13“}}} |Where {$_.Network-ne “”}

And it will output the VM name and the Network Adapter causing the conflict. :)

| More

How to install an SSL cert chain on a Cisco Wireless Lan Controller (WLC)

June 2nd, 2010 Jason Samuel No comments

Cisco WLC (Wireless LAN Controllers) require the entire SSL cert chain to be installed on the appliance. I like to do this via IIS since it is very quick and efficient and all my intermediate and root certs are already on there typically (double check your servers please).

1. Create a temporary site in IIS and generate a CSR. Make sure the domain matches whatever appliance you are trying to generate it for exmp: wireless.yourcompany.com.

If you are exporting a wildcard cert that might already be on your IIS server, just export it as a PFX and skip to step 5.

2. Send the CSR to your registrar and wait for them to validate.

3. Get the .crt and convert it to .cer (just open the .crt and save it as .cer). Do the same for the intermediate and root certs you received.

4. Install the .cert in IIS as well as all intermediate and root certs they send you.

5. “View” the cert in IIS (Site properties > Directory Security > View Certificate > Details Tab > Copy to File) and export it with the private key in pkcs12 (.pfx) format. Also make sure to check “Include all certificates in the certification path is possible”. Just set the password as “password” so it’s easy to remember.

6. Now you have a .pfx with your whole SSL cert chain but you need to convert it to .pem for the WLC.

7. Use this command with OpenSSL:

openssl pkcs12 -in yourcert.pfx -out yourcert.pem

8. Type the “password” when prompted

9. TFTP the new .pem containing the whole SSL cert chain and private key over to the WLC and reboot. You will have to specify the “password” when you type in your TFTP info and Save and Reboot the WLC after the download is successful. Test it out and it should work.

| More