JasonSamuel.com

I’ve covered Access Gateway quite a bit in the past and these two articles on my blog are a good primer for what I’m about to cover:

1. How to setup your Citrix Netscaler (Access Gateway) and Web Interface for iPads and mobile devices that use Citrix Receiver

2. Quick way to brand Access Gateway and Web Interface 5.4 with company logo and colors

Now let’s throw a new scenario into the mix. You want to allow users from another domain access to resources through your Access Gateway and Web Interface. Here is what you need to do to accomplish this.

SETTING UP A DROPDOWN BOX TO PASS THE DOMAIN
On your Access Gateway, if you followed my “branding guide” article above you will have noticed on the themes available from the Citrix blogs there are 2 versions, one normal and one with a dropdown. If you chose Horizon to customize like I did, make sure to download Horizon2.gz which has the code for the dropdown already in it. You just need to add your domains in the code. This is found in these two files:

\Horizon2\ns_gui\vpn\index.html

\Horizon2\ns_gui\vpn\tmindex.html

If you have your own theme and just want to add the code, Citrix has an excellent CTX article here covering it:

How to Add a Drop-down Menu with Domain Names on the Logon Page for Access Gateway Enterprise Edition:
http://support.citrix.com/article/CTX118657

What’s happening here is that a dropdown box is created with pre-populated domain names that you specify in your code. The user selects the proper domain when logging in and then this domain name is inserted into the header and passed to the Netscaler Access Gateway vserver where the authentication policy examines it. Based on the name the Netscaler sees in the header, it forwards the request on to the correct authentication server. This works great with LDAP authentication servers. I have not tested it with TACACS or RADIUS yet.

SETTING UP ACTIVE DIRECTORY LDAP AUTHENTICATION
Now it’s time to setup your Active Directory authentication for each of your domains.

1. Download LDAP Browser 4.5 for free here: http://www.ldapbrowser.com/download.htm

2. Go to your Netscaler under System > Authentication > Servers tab and create a new LDAP server. Use port 636 for secure LDAP instead of 389. Use LDAP Browser to get the Base DN and Administrator Bind DN. I prefer to use the app so I don’t fat finger the distinguished name and spend an hour trying to figure out that I missed a comma. Using LDAP Browser is much quicker and less error prone. The Base DN can be your entire AD domain or you can lock it down to an OU. The Administrator Bind DN is the actual account name that is used to enumerate AD. So CN = the user name, OU= the OU it’s in, etc. Add the password for this service account.

Make sure to select “SSL” for security type near the bottom. Click the blue “Retrieve attributes” link and it should talk to AD. Then the drop down boxes in Other Settings will not be greyed out anymore. Set the following:

Server Logon Name Attribute = samAccountName
Group Attribute = memberOf
Sub Attribute Name = CN
SSO Name Attribute = UserPrincipalName

So everything should look like this so far:

3. Now expand the Netsted Group Extraction arrow near the bottom. You need to enable it and set the same Group Name Identifier as you did for Server Logon Name Atrribute. In this case samAccountName. Then press OK to create the server:

4. Now it’s time to create the policy and bind it to the server. Hit the tab for Policies and create a new policy. Choose the server you just created. For the Expression, you will want:

REQ.HTTP.HEADER Cookie CONTAINS domain1

where domain1 is the name of your first domain.

5. Now do steps 2 through 4 again but for your second domain, domain2.

6. Almost done. What we’ve done so far is create two authentication servers and two authentication policies. These two policies are good for web browser users (IE, Firefox, Chrome, Safari, etc.) but mobile devices using the Citrix Receiver native app like iPhones, iPads, Android phones, Android tablets, etc. will not work just yet. This is because they cannot pass cookie values. Nor do they pass the domain during authentication against the Access Gateway (a traffic capture can confirm it only passes user ID and password). You need to create a different set of authentication policies for these. So you will create one for each domain. You will bind to the same authentication server as the other policies but you’ll want to add the word “mobile” or something descriptive to the end of the policy name so you know what it is for. The expression should be:

REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver

7. Now do the same for your other domain, domain2.

8. Now go to your Access Gateway vserver and add all 4 policies you created. Take a look at my example screenshot closely and the priority order.

Here is the logic behind this policy order:

100 – Domain1 user logs in via web browser and uses drop down/passes cookie for domain1 so authenticates against domain 1 DC.

110 – Domain2 logins via web browser and uses drop down/passes cookie for domain2 so authenticates against domain 2 DC

120 – Domain1 user logins via iPad so first two policies are automatically false since the native Citrix Receiver app can’t pass a cookie. Authentication policy is looking for CitrixReceiver in the header so it will try to authenticate against domain 1 DC. It will be successful and the user will see his apps.

130 – Domain2 user logins via iPad. The first two are false for the same reason as 120. The expression from policy 120 is true so the user will actually authenticate and fail against domain 1 DC. Then it moves on to the 130 policy and authenticates against the domain 2 DC. It will be successful and the user will see his apps. Note, if a user has the same account name on both domains, it will cause a rejected login attempt on domain 1 which depending on your Active Directory account lockout policy can eventually lead to an account lockout. Just be mindful of this and adjust your policy accordingly if you need to.

MODIFYING YOUR SESSION POLICIES
9. Now on to the session policies. If you followed my guide on How to setup your Citrix Netscaler (Access Gateway) and Web Interface for iPads and mobile devices that use Citrix Receiver you should have 2 policies. One for regular traffic and one for mobile/Citrix Receiver traffic. Should look something like this:

Go ahead and click on the blue profile link for the CitrixReceiver policy first.

10. Under the Published Applications tab, clear the Single Sign-on Domain, you don’t need it anymore. SSO is being handled by the authentication server now. Specifically the SSO Name Attribute setting for each server which you set as UserPrincipalName. So it is automatically passing the UPN formatted user credentials to the Web Interface as the same time the user is successfully authenticating against the Access Gateway using the SAM account name. Pretty cool right? Now do the same for your other policy that controls regular web traffic.

11. An extra step for your mobile traffic policy. Under the Client Experience tab, you can set the Clientless Access to Allow and the Plug-in Type to Java but it’s not necessary. It will still work but since it’s mobile traffic only, this won’t hurt.

SETTING UP ACTIVE DIRECTORY AUTHENTICATION GROUPS
12. Now let’s move on to setting up your authentication groups. Let’s say each domain has a security group that gives users access via the Access Gateway. Again, there are 2 well documented methods for group extraction:

How to Configure a NetScaler Appliance for Active Directory Group Extraction for LDAP:
http://support.citrix.com/article/CTX111079

How to Configure a NetScaler Appliance for Active Directory Group Extraction for LDAP Using the Groups Allowed To Login Feature:
http://support.citrix.com/article/CTX125797

I prefer the first method. It’s basically just one step in one place (the authentication server object) and goes into effect at the authentication level. No fooling around trying to control it via a session policy. So go back to your 2 authentication servers you created before and take a look at the Search Filter field. It should be blank at the moment.

13. Go to LDAP Browser and get the DN for for the security group you want to give access to for your first domain.

14. Go back to the Search Filter field and type memberOf= followed by the DN (distinguished name) for the security group. So it should look something like this:

That easy. Make sure to do it for the authentication server for domain 2 as well. Go ahead and test your login scenarios and everything should work perfectly.

ADDING REDUNDANCY FOR YOUR AUTHENTICATION SERVERS
I only covered how to add a single authentication profile (domain controller) for each of the 4 policies on the vserver. Most companies are going to have multiple DCs. There is no way to add multiple DCs per authentication policy. The solution is to create a Load Balanced vserver with all your DCs behind it per domain and then add the IP of the vserver to an authentication server that is bound to the authentication policy. So in the example I’ve been showing you, I would have to create 2 load balanced vservers since I am working with 2 domains.

15. Go to Load Balancing > Servers > and add all your servers here:

16. You guys know I prefer to create service groups vs. services and the reasons why if you’ve read my previous Netscaler articles so go to the Service Groups section and add a new services group. Select SSL_TCP for the protocol.

Specify the server based members and make sure to set the port to 636. You can weight the servers here if you want to. Under the Monitors tab, go ahead and add a ping monitor or whichever monitor you prefer. Do this for each group of your authentication servers. You might want to group them by city, datacenters, production vs. DR, or whatever makes sense in your environment.

17. Now create a load balanced virtual server for your first domain. Select SSL_TCP for the protocol, port 636, and give it an IP address. Under the Service Groups tabs, add the groups you need. Keep in mind you won’t be able to see the SSL_TCP service group you created until you set the vserver protocol to SSL_TCP. Under the Methods and Persistence tab, you can get creative if you like or just leave it the default Least Connection. It really depends on your environment and where you prefer traffic to go. Under the SSL Settings tab, bind the cert you use for your Access Gateway vserver. Do all of this for your other domain as well. Both load balanced vservers should be in the UP state at this point.

18. Now go to System > Authentication > Server tab and left click on one of the DCs. Then at the bottom click Add, this will copy all of the settings on the DC you had highlighted and allow you to create a new one based off of it. So you don’t have to put all your DNs and stuff in again. Just give it a unique name and for the IP, type in the IP of LB vserver you just created for that domain:

Now do the same for the other domain.

19. Now go to the Policies tab and edit each of your 4 policies. You can simply change the server in the drop down to the new one you created for each of your 4 policies and you’re done. All authentication traffic is now going through your load balanced vservers. Keep in mind that before, LDAP traffic was going from your NSIP to each DC. When you use LB vservers, traffic is going from the SNIP to each DC. So make sure you have your firewall ports open from the correct source IPs or you won’t be able to authenticate. Retest all your scenarios and everything should continue to work just as before.

TROUBLESHOOTING LOGINS
Once you’ve set everything up you might encounter issues and need to troubleshoot. There are a few ways to troubleshoot logins. I’ve covered this before here:

How to troubleshoot RADIUS or TACACS authentication issues on a Netscaler/Access Gateway

20. First you can troubleshoot the authentication layer but capturing of all authentication happening on the device. Open Putty and connect via SSH to your Netscaler. Enter the shell by typing

shell

and pressing enter. Then type:

cat /tmp/aaad.debug

and the cursor will go to the next screen and wait. Open up your web browser and attempt to login to the Access Gateway. Immediately you will see the request happen in your SSH window. You’ll see exactly what is happening line by line like this. You’ll even see the nested group extraction taking place:

The last line will show the accept or reject:

When you’re done, press Ctrl+Z to exit.

21. The next layer to troubleshoot is all the polices. All of them, authentication, session, etc. You want to see everything a user hits when he logs in. So type the following:

nsconmsg -s disptime=1 -d current -g pol_hits

and login to the Access Gateway again using a web browser. Immediately in your Putty window you’ll see what all policies were hit:

This will help troubleshoot your policy flow. We’ve built a lot of intelligence into the cascading authentication and session policies and based on the user’s device and domain, it will vary what all they hit. This is an excellent way to test each of those scenarios.

22. The last step is getting down to the packet layer and analyzing the traffic using Wireshark. I’ve covered this in my previous post but go to System > Diagnostics > Start New Trace and set the packet size to 0. Then press Start.

Login to your Access Gateway via web browser, stop the capture, and download it to your desktop. You will also want to WinSCP into the Netscaler and grab the SSL RSA key for the SSL cert you are using on your Access Gateway site so you can decode the SSL traffic. Then double click on the .cap packet capture file and it will open in Wireshark. Go to Edit > Preferences > Protocols > SSL > and click Edit in the RSA Keys field:

In the SSL Decrypt Window, hit New in the bottom right side and this window will popup. Type in the IP address of the Access Gateway vserver, port 443, protocol http, the exact path on your hard drive to the SSL RSA key file, and leave the password blank. Then press OK to add it:

Press okay and get back to your capture. Now in the Filter field, just type the following:

frame contains youruserID

because we want to find all the instances of your user ID being passed. You should be able to find your user ID and password in plain text. Follow the TCP stream if you need to to find out exactly what is happening.

Hope this helps. Please leave a comment if this post helped you or if you have any questions and I’ll try my best to help.

Today I would like to go over proper URL redirection when using SSL but first I would like to preface this by describing what proper URL redirection is. Anytime you inherit, consult for, or build an environment out from scratch, you must always verify every vector a user may use to get to your website. You must plan accordingly to intercept and redirect the user transparently when they manage to get to URLs that you don’t want them to use. They should never be prompted for an action (such as “click here” to continue) or receive a warning/error message if at all possible.

A 301 redirect is in my opinion to best way to steer a user to where they need to go. A 301 tells the browser/search engine spider that the redirect is permanent. In the case of spiders, a 301 will tell it only the destination URL should be indexed. A 301 redirect works fine over regular HTTP connections and all web servers and load balancers are capable of performing them. Alternatively, you can also use a 302 redirect but this tells spiders the redirect is temporary. Not very good for SEO purposes if your organization is concerned about that.

The problem occurs when you try to do a 301 or 302 redirect to an SSL URL (HTTPS URL) but the SSL certificate for that URL does not match the domain. It will throw a warning message saying the connection is untrusted or that there is a problem with the website’s security certificate in the user’s browser. Most users will come across on an HTTP connection such as http://www.domain.com and if needed, the network/web engineer would setup a web server or load balancer policy to perform the 301 redirect to https://www.domain.com. The certificate in this case is valid for “www.domain.com” and will not throw an error. But what if the user somehow manages to go directly https://domain.com over an SSL connection? It would throw an error unless the certificate is valid for “domain.com” which is most cases it is not. Even if you had a wildcard certificate for “*.domain.com”, it would not prevent you from getting a warning because this specifies any subdomain is valid, but does not cover just the domain by itself. Your user will always get a warning message. You may think you could write a policy on your load balancer to do the redirect but that will not work. The reason is a TCP connection has to be established first and during the SSL handshake before the connection is fully established is when the user gets the warning. The policy will never run as a result until after the user accepts the warning message and the SSL handshake is completed.

There are a couple of ways to get around this. The cleanest way of doing it is using a Subject Alternate Name (SAN) SSL certificate. A SAN cert allows you to specify exactly what domains you would like the certificate to be valid for. The price on SAN certs is much higher than a standard certificate or even a wildcard certificate and they are only available from a handful of certificate authorities at this time. They require a lot more validation typically and the validation term is usually a bit lower than a standard certificate.

Let’s compare a SAN SSL certificate vs. a Wildcard SSL certificate. A single SAN cert would cover me for the following if I wanted it to:

domain.com
www.domain.com
www.myotherdomain.com

while a single Wilcard SSL cert would only be able to do the following:

*.domain.com

It is possible to combine both so you could get a SAN + Wildcard SSL certificate that will cover all of the above. To my knowledge, the only certificate authority that currently offers them is DigiCert. They have a pretty nice write up about their cert here:

http://www.digicert.com/ssl-support/wildcard-san-names.htm

I once asked a VeriSign managed PKI rep why they don’t offer wildcard + SAN like DigiCert and he went on for 5 minutes about ICAAN policy. Why one cert authority can issue while another won’t, I am guessing it likely has to do with acquisition costs for them. You have to remember, VeriSign is still trying to convince people to buy SGC SSL certs (for legacy browsers) at a premium price when in my opinion, SGC has been obsolete for years now and any web/network engineer that knows about SSL encryption has anything below 128 bit locked down anyhow. But I guess if people will buy it, they will sell it. Either there just hasn’t been enough demand or their costs are too high for wildcard + SAN certs. Not trying to pick on VeriSign, just making an observation. I still purchase certs from them regularly (non-SGC of course). Most every large organization does, but sometimes the wildcard + SAN might be the best solution for you. I know some CDN providers like Akamai sometimes have a need for them and purchase through DigiCert.

Getting back on subject, a second way of getting around this is purchasing two standard SSL certifcates and putting them on different IPs. One will do a 301 redirect to an another. So you would have to purchase an SSL cert for domain.com AND www.domain.com.

Let’s look at some real world examples to better understand this. I am going to compare Chase, Bank of America, Citrix, and Amazon.com to show the different types of solutions for this problem.

Chase.com
Chase uses SAN certs from VeriSign. Again, SAN certs are a very pricey approach but the easiest. Your configuration on your load balancer and servers is minimal since one SSL cert will take care of everything. This will result is less overhead on your load balancers as well which is a big plus when you serve up millions of connections.

Go to https://chase.com, it will immediately do a 301 redirect to https://www.chase.com. Here is a screenshot showing the 301 redirect to https://www.chase.com and the certificate which shows the 4 Subject Alt Names Chase’s cert is valid for:

As you can see, the certificate is valid for www.chase.com, chase.com, www.yourchasefreed.com, and yourchasefreedom.com. You will also notice that the cert is only valid for a 1 yr period. Typically CAs will try and limit you to a max of 2 years on SAN SSL certs and EV SSL certs but it is always best practice to keep them to 1 year just in case your cert keys were compromised. A SAN cert can have many domains leaving you very exposed, so it is a good idea to change it every year for this reason.

Bankofamerica.com
Bank of America uses standard SSL certs from VeriSign. Cheaper but a little extra configuration involved on the load balancer. Let’s take a look at what happens when you visit https://bankofamerica.com:

You will be 301 redirected to https://www.bankofamerica.com just as you were with Chase. But this time using a standard SSL cert with one CN name as you saw above. You can verify this by pinging bankofamerica.com which resolves to 171.159.100.173 and typing just the IP into the browser over SSL as https://171.159.100.173. This will give you an SSL error before the 301 redirect gets a chance to run and you can view the cert:

So just to be clear, typing https://bankofamerica.com will use the above cert which is valid, will complete the connection, run a policy that will do a 301 redirect to https://www.bankofamerica.com, cert for https://www.bankofamerica.com is validated, then the connection is completed and the page is served up. A perfectly good way around the problem using standard SSL certs as opposed to SAN SSL certs.

Citrix.com
Citrix is another example of using the standard SSL certificate redirect method much like Bank of America. But they do not leverage the 301 redirect at all. Instead they use a 302 temporary redirect pointed directly to their index page. The 302 is likely done for SEO purposes so the “www.citrix.com” domain is indexed in search results and not the “www.citrix.com/lang/English/home.asp” destination page. In this case, a Netscaler with either a responder policy or content switching policy will send you to https://www.citrix.com or https://citrix.com which both display the same content. They are on different VIPs with different SSL certs. One is a standard cert while the other a wildcard. They could have used two standard certs if they wanted to as well of course but in this case, they may use the wildcard for other subdomains, GSLB, etc. and the standard cert is meant to solely capture those users typing in an incorrect URL:

Amazon.com
Woops! The world’s largest online retailer does not seem to be redirecting customers that arrive on an irregular URL over SSL properly. If you go to https://amazon.com, it will give you a “This Connection is Untrusted” error. It’s not the end of the world but still, a user should never see a message like this.

In IE for example, it will say “There is a problem with this website’s security certificate”. Again, not the end of the world. But in Chrome, the error will have a bright red background and say “This is probably not the site you are looking for!
You attempted to reach amazon.com, but instead you actually reached a server identifying itself as www.amazon.com. This may be caused by a misconfiguration on the server or by something more serious. An attacker on your network could be trying to get you to visit a fake (and potentially harmful) version of amazon.com. You should not proceed.”

Yikes! Do you really want your users to see a message like this?

Interesting to note however, going to https://www.amazon.com does a 301 redirect to http://www.amazon.com. Obviously Amazon wants the customer to use HTTP rather than SSL on their homepage. Much less overheard on their load balancers plus they don’t have a login box directly on their home page so no need for SSL to begin with. This is thought out well but I am puzzled as to why they don’t do anything about https://amazon.com. As you saw with Chase, Bank of America, and Citrix, the different solutions are very easy to implement.

I hope this article has been helpful. Please post a reply if you have a different solution to the problem that I did not cover. I would love to hear your insights.

Citrix is now offering Netscaler VPX Express licenses with throughput increased from 1 Mbps to 5 Mbps! This will help tremendously in test lab environments for a lot of people. Most of my important test environments where I need some extra bandwidth were on VPX 10 or VPX 200 licenses already but now with a 5 Mbps throughput VPX Express for free, it gives me a little more wiggle room before I need to shell out the cash for these licenses.

That’s not all! Now those of us that are existing Netscaler customers are entitled to VPX Developer edition licenses with Platinum edition features! This makes testing features like integrated caching for example easier without having to pay for a license first. Sadly you are limited to 1 Mbps but it should be more than enough to test low bandwidth apps against Platinum features. You can always get a 90 eval though which has a 1 Gbps limited if you really need the extra bandwidth for testing.

Good move on Citrix’s part! This how how you corner the application delivery market and then stay on top. Give it away for free and let people see how powerful the Netscaler platform is first hand.

Read more about the changes on the Citrix Community blog here.

In a worst case scenario and all your web servers have failed, what do you do? You could have a standby group of servers or CDN on or off premise to pick up the load or at least display a maintenance page but this is worst case scenario. A catastrophic failure and ALL your servers are down due to a code issue, server configuration issue, database issue, virtual infrastructure failure, SAN failure, maintenance being performed on all servers at once (I hope not on purpose), virus outbreak, or whatever else kind of horrible scenario you can think of. You get traffic all the way up to the Netscaler appliance but since your vserver is down, the user’s browser will timeout as if your company fell off the face of the earth. This is very unprofessional for any organization. Users timing out or seeing a “page could not be displayed” error is unacceptable.

So the solution is to have the Netscaler display a maintenance page with the code hosted on itself somehow. I tried several different methods including content filtering and responder policies using HTML. Originally I even thought I could leverage integrated caching to serve up cached pages and static content like images. I settled only using a responder policy initially which worked. Citrix even has a very nice knowledge center article (CTX117337: How to Configure a Maintenance Web Page by using the Responder Feature of the NetScaler Appliance) which is located here:

http://support.citrix.com/article/CTX117337

In a nutshell, what the author of the article wrote is basically more or less the same conclusion I reached as well. I just did it via GUI and that is what I will show you below. But I was not happy with the result. Keep reading and you will see why. FYI, I did all the screenshots below on an NS 9.1 appliance but it is the same procedure on NS 9.2 or any other version.

1. I am going to assume you have servers, services/service groups, and a vserver already that is UP and running. I will call them the following in this example:

vserver – lb_vsver_mywebsite
service group – svcgrp_myservicegroup
server – svr_mywebserver

Excuse the redactions in my screenshots please, I had some other configurations on this test appliance and I don’t want to confuse you with it:

vserver:

service group:

server:

2. Now create a backup vserver for your existing live vserver. In this example, I have called it “lb_vsvr_bkup_mywebsite”. But instead of giving it an IP, just uncheck directly addressable. This will cause the IP area to become greyed out:

backup vserver:

When you click Create, it will show up as running on the IP 0.0.0.0 like below:

3. Now you need to create a service that is always UP and bind it to this backup vserver so that it will always remain UP. Just go under Load Balancing > Services, and click Add. Then create a service called “svc_maintpage” but for the Server, type in the localhost IP of 127.0.0.1, add a ping monitor, and press create.

4. Now go back to your backup vserver and bind this new service to it. Immediately after clicking OK, the backup vserver should go into an UP state. You might need to refresh your window if it doesn’t.

5. Now double click on your live vserver and under the Advanced tab, choose “lb_vsvr_bkup_mywebsite” for the Backup Virtual Server option and press OK:

6. Now under Responder > Action, click Add to create a new action. This is where you get to put some HTML and CSS. It must be very basic, all parenthesis have to be removed when using CSS in the HTML body or it will give you can error, and the whole policy must be under 255 characters total. I will name mine “action_mywebsite_maint_page” and here is an example of my policy I will use with it:

"HTTP/1.0 200 OK" +"\r\n\r\n" + "<html>
<style type=text/css>
<!--
.mywebsitefont {
  font-size: 24px;
}
-->
</style>
<body class=mywebsitefont>Sorry, our website is currently not available. 
Please try again later.</body></html>" + "\r\n"

7. Now under Responder > Policy, click Add to create a new policy that will call on the action you just created. In this example, all we need is for the HTTP request to be valid and we will display the maintenance page. I will name it “resp_policy_mywebsite_down” in this example. Choose the action you just made in the Action drop down and for the expression, just put:

HTTP.REQ.IS_VALID

8. Now go back to the Load Balancing folder and double click your backup vserver and bind the responder policy to it like below:

9. Now to test. Open up your website in a browser and it should display as normal right now. Now login to your webservers and turn off your websites. Immediately your live vserver should say DOWN for the State but the Effective State should remain UP. This is because all traffic is being forwarded to your backup vserver you specified earlier which is set to always be up:

Refresh your browser and you should now see the maintenance page you created like below:

As you can see, a simple HTML page like above is not very professional. We need more HTML/CSS than 255 characters to work with and we need images working to make it look professional. At least it is better than a page timeout though!

Now with a content filtering policy, you don’t have to worry about a character limit. You can get away with putting HTML/CSS in a content filter policy. But again, where do the images come from?

I decided to call Citrix and see if they have run into a request like this. They had not. Now off the bat both techs I spoke to said what I was trying to do is not supported by Citrix. A Netscaler is not designed to do this. But luckily the second tech Brian at Citrix Support was just as enthusiastic about getting something to work as I am and wasn’t going to give up easily so we went over a few scenarios. The Netscaler does have an Apache web server on board, that is how the admin GUI is display to you. It is also how the Access Gateway portal is displayed to the end user. We needed to figure out a way to leverage the Apache web server on board the Netscaler to host our images, HTML, CSS, etc. The initial thought was to overwrite the Access Gateway portal and create a responder policy that would do a redirect to an Access Gateway vserver you create. The negatives here are that you are limited to SSL traffic only, have to worry about having a valid cert, you can’t bind all the policies you might need to it like you can a load balanced VIP, etc. I didn’t feel that comfortable destroying functionality to gain other functionality either.

In the end, the solution was easy and did not require overwriting the Access Gateway portal. We can host our HTML, CSS, and images on the Netscaler itself and point Apache at it. Brian did a quick proof of concept in his lab. Then I improved on it a bit. Here is the end result which I am sure a lot of you will find pretty handy in your organizations. Steps 1 through 5 are the same as above. Then from there, begin these steps:

1. First we need to get our HTML, CSS, and images on the Netscaler. WinSCP into your Netscaler and go to “/netscaler/ns_gui”. The folders you see called admin_ui, vpn, etc. are what host the Netscaler Admin GUI and Access Gateway respectively. So you have the option of putting something in the root of this folder or even create a separate folder here if you want. In my case, I decided to put a “maintenance.htm” in the root and also create a folder called “static” that will host most static content like CSS and images.

2. Now under Responder > Action, click Add to create a new action. Very important, make sure to change the type from Response to Redirect. The action should be the following (with parenthesis included):

"http://www.mywebsite.com/maintenance.htm"

3. Now under Responder > Policy, click Add to create a new policy that will call on the action you just created.. Your responder policy will need to allow the maintenance page, plus CSS, .gifs, and .jpgs you might use. So the policy I will use is:

!HTTP.REQ.URL.CONTAINS("maintenance.htm") && !HTTP.REQ.URL.CONTAINS(".gif") && !HTTP.REQ.URL.CONTAINS(".jpg") && !HTTP.REQ.URL.CONTAINS(".css")

4. Now go back to the Load Balancing folder and double click your backup vserver and bind this new responder policy to it like I did below:

Now if you disable your service groups and check your maintenance page again, you can see how the website displays the full page with nice HTML, CSS, and images. In this example, I borrowed the Sears.com maintenance page. Notice how showing your company logo keeps your branding intact even on a maintenance page which is the correct way to handle a website issue. Tell your users you are aware of the problem and offer alternatives in the meantime (static links along the bottom to other servers that are up and offering content in this example). You don’t have to go that far but it’s always nice to let your user base know you haven’t disappeared and your infrastructure is solid. This is very professional and above all, automated!

The only problem here is that when your website is back up, users will still be refreshing on this maintenance.htm page. They will get a 404 error. So you have four options. I usually prefer number 4 personally but it all depends on your needs:

1. Change your maintenance.htm page to say index.htm or whatever page is the default page of the root of your website so when they refresh once the vserver is back up, they will get the live page. You will need to WinSCP into your Netscaler again and change the maintenance.htm file name as well as change it in your Responder Action. The issue here is if let’s say you are using .NET, you can’t call it index.aspx because Apache on the Netscaler can’t parse it.

2. Just create a link on the page that says “Click Here to Try Again” which is pointed at the correct index page. This assumes the end user will actually click the link instead of hitting refresh. You can’t be 100% sure they will do this.

3. Create a maintenance.htm page on your servers and then set IIS, Apache, or whatever web server you use to do a 301 redirect to your live index page. You can leverage the Netscaler to do the redirect too of course.

4. My preferred method. Create a new responder policy saying any maintenance.htm should automatically redirect to index.aspx and bind it only to your real vserver. That way anyone that requests that page when your servers are up will always be redirected to your index page. In this example, I will call my live site’s index page index.asp and call the action policy “action_mywebsite_index_redirect”. I will also make it redirect to SSL in this example because there is a login box on the index.asp page and I want to keep it secure using https:

I will call the responder policy “resp_policy_index_redirect” and for the expression, tell it to redirect any requests to “/maintenance.htm”:

HTTP.REQ.URL.CONTAINS("/maintenance.htm")

Now bind this to your live vserver:

Now you can test it by disabling and enabling your servers or service groups. It should transition automatically between your maintenance page and the live index page.

One thing I would like to point out. On any of your Responder Policies or Actions, you can always view the hit counter to see if the policy or action is being invoked. This might help you when you are setting this up initially and something goes wrong and you want to see if the policy or action is being hit:

So there it is. Your Netscaler is now an emergency web server that automatically puts up a professional looking maintenance page in a worst case scenario when every backend web server you have is down. A big thank you to Brian at Citrix for the help! If anyone can think of any improvements to this process or has any trouble with it, please reply I would love to hear about your experience.

I get asked this question a lot. How do you tell what license you have applied to a Netscaler VPX appliance? The next question is what edition VPX do I have: Standard, Enterprise, or Platinum? Most people expect either one of these very important pieces of info would be displayed right along the top header where the IP and hostname are or at least under the main System window under the Hardware Information section but it is not. You can figure out what version you have by looking at the features of the appliance and looking at the current throughput to deduce the license and edition but that is not very practical. There are 3 ways you can see the current license applied easily without having to play detective:

1. Click Licenses under the System folder and the last item in the first column is “Model ID”. For physical devices, it will show the model number like 7500, 9500, 15000, 17500 etc. But for VPX appliances, it will say 1 for the the free VPX Express appliance (denoting the 1 Mbps limit) and 10, 200, 1000, or 3000 for the pay for licenses. If you just upgraded from the free version to a pay for version, you can verify it was applied here at a glance easily.

2. WinSCP down the license file from the appliance and open it in Notepad or any text editor. Make sure wordwrap is off. The third line of the license file will tell you the version. For example, something like this:

#CITRIXTERM FEATURE 1.0 CNS_V200_SERVER EN Citrix NetScaler VPX 200|Server

This means the license I have is for a VPX 200. If you take a look at the very first line and scroll to the right some, you can see the edition level info under INCREMENT on physical appliances:

CNS_SPE_SERVER

The SPE part means I have a Platinum Edition license. Standard Edition would be CNS_SSE_SERVER and Enterprise Edition would be CNS_SEE_SERVER.

On a VPX virtual appliance, it will never say this. Instead it will say something like:

CNS_V200_SERVER

which tells you it is a VPX 200, but you have no way of telling the edition. The only thing you can do to figure out what edition it is by looking at the features available to you and comparing to the the VPX feature matrix here or move on to option 3 below.

3. Login to MyCitrix.com and look at the license file there under Manage Assets. It will tell you which version you have. But when you have 50-60 Netscalers you might be responsible for, this becomes a bit tedious to wade through. You could also call Citrix Support and give them your license number and they can tell you what edition your license is but then you have to play the waiting game on the phone with them. Easier to just login to MyCitrix.com and go through your licenses.

Hope this helps. Please do post a reply if you have an easier way of identifying your VPX license edition level at a glance.

I recently received a request to catch all requests on HTTP and redirect the user to the exact same URL that was requested but over SSL (HTTPS) instead. You can do this using a Responder Policy.

Basically the action to redirect over SSL will be:

"HTTP/1.1 301 Moved Permanently\r\n" + "Location: https://" + HTTP.REQ.HOSTNAME + HTTP.REQ.URL.PATH + "\r\n"

Make sure to Bypass the Safety check or it will give you an Input Expression is unsafe error. Don’t worry, this is just a warning message. The Netscaler tries to resolve URLs to make sure they respond when you write an expression. In this case we are telling it to catch any URL. So since it can’t validate something general like this, it throws the expression error. Make sure to bypass the safety check to get around this.

The responder policy that calls the action by looking for non-SSL connections should be:

!CLIENT.SSL.IS_SSL

You will notice I specified a 301 redirect in my action because I want to make sure any search engine spiders index my site URL as “https://www.mywebsite.com” and not as “http://www.mywebsite.com” for SEO purposes.

You may or may not want to bind the policy globally. I only apply responder policies to vservers that need them just to be cautious.

There are 4 easy ways to check that SSLv2 and weak ciphers are disabled on your web servers and appliances. SSLv2 is deprecated and should never be used. You will definitely need to verify these are disabled for PCI compliance and SOX compliance. The instructions below apply to check SSLv2 on any web server (IIS, Apache, iPlanet, etc.) or any traffic or load balancing appliance (Citrix Netscaler, F5 BigIP, etc.).

1. SSLDigger - Download it here http://www.foundstone.com/us/resources/proddesc/ssldigger.htm. It’s a great little tool that first tests for SSLv2, then gives you the option to test for weak ciphers one by one, then spits our a nice little HTML report of the findings. You can even create a .txt file with all your URLs to test and perform testing in bulk. This is wonderful when you have many domains to test and need to give a report to management quickly.

Checking the Google.com SSL cert (click to enlarge):

Checking for weak ciphers systematically (click to enlarge):

HTML report (click to enlarge):

2. OpenSSL – The old favorite. Type the following to connect to your web server or appliance over SSLv2:

openssl s_client -connect www.yourdomain.com:443 -ssl2

It should connect giving you a CONNECTED(000007C0) message but then immediately fail and display an error message like one of the following:

419:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:
420:error:1406D0B8:SSL routines:GET_SERVER_HELLO:no cipher list:s2_clnt.c:450:
10076:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:. \ssl\s2_pkt.c:428:
write:errno=548965:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:
write:errno=10054

SSL handshake failures and connection resets (errno=10054) are completely normal if SSLv2 is disabled. The specific error message you get will depend on exactly what behavior your web server or appliance is configured to do when a client connects using it.

Example of SSLv2 disabled:

Example of SSLv2 enabled (VERY VERY BAD!):

3. Nessus – Download it here http://www.nessus.org and use the Home Feed and setup a new scan using the SSL Anonymous Cipher Suites Supported and SSL Version 2 (v2) Protocol Detection plugins. Please make sure to disable all the other plugins, you don’t want all that hitting your server. Just these 2 are sufficient. Then check the Reports and see if it is reporting SSLv2 as a vulnerability.

4. External SSL checkers – Running nice little tools like the above three is nice to check your servers when you are internal but if you want to see what 3rd parties are seeing externally, use one of these online SSL checkers to test:

http://foundeo.com/products/iis-weak-ssl-ciphers/test.cfm
http://www.dnsstuff.com/tools/

I’ve covered how to convert an SSL cert to a .pem with a private key for a Netscaler before here but sometimes you might be required to install a full SSL cert chain (read my article here on how SSL certificate chains work). I actually install the full SSL chain (root and intermediate certs) on any Netscaler I deploy as a best practice.

I’m going to use Network Solutions again in this example but this will work for Verisign, GeoTrust, Thawte, Comodo, or any other CA (certificate authority).

Here is how you do it on the Netscaler:

1. I’m going to assume you have already generated a CSR (either through the appliance or IIS, whatever your preference), had your cert validated, and have downloaded your site cert and intermediate/root certs from Network Solutions since I have covered this before in the articles I mentioned above.

2. Now just login to your Netscaler, install your website cert that was issued to you first under SSL > Certificates on the Netscaler.

3. At this point, if your website is publicly available, let’s examine the cert chain. Go to http://www.sslshopper.com/ssl-checker.html and type in your website (if it’s not publicly available, use OpenSSL which I have written about here). Since you have not installed intermediate certs on your Netscaler yet, it should report your SSL cert is good but your SSL cert chain is broken like below:

4. Now go back to your Netscaler and install the two intermediate certs and one root cert that Network Solutions has give you. You really only need the intermediate cert but I like to install them all just in case. They are in .crt format when you get them from Network Solutions and that is fine, no need to convert. Just go to SSL > Certificates on the Netscaler and click Add in the bottom toolbar. Give it a name and use Browse (Local) to find your certs and install all 3 one by one. Here is what it should look like when you are done:

5. Now here comes the most important part. You need to link the intermediate cert (NetworkSolutions_CA) to your site’s SSL cert. Right click on your site’s SSL cert and hit “Link”. Now choose the NetworkSolutions_CA cert in the drop down box and press OK. It will confirm the cert is now linked. The Netscaler actually supports up to a 10 cert chain so you can have your website cert + 9 more intermediate certs linked to it if you want. So feel free to link more if you need to at this point. You can also click on the “Cert Links” button in the bottom toolbar and it will show you all your links. For example, here is a Netscaler with 3 certs from Network Solutions that have been linked to the “NetworkSolutions_CA” cert:

6. Now go back to http://www.sslshopper.com/ssl-checker.html and try your domain again. This time it will say your SSL chain is good:

Here’s an example where I linked each intermediate cert chain all the way down to the root certificate. I would recommend you link all your certs down to the root as a best practice:

Also, Citrix put together this really good guide on SSL cert chains I am hosting here for you. It’s a great read with plenty of screenshots so take a look:

Citrix_Verisign_NetScaler_SSL_Cert_Deployment_Guide

If you work with Netscalers, you already know how SNIPs and MIPs work. They are the IPs that communicate with your servers. Every packet from the outside world that passes through the Netscaler will hit your server as though it was coming from this IP. This is the default behavior on a Netscaler.

Unfortunately if your servers require the client’s true IP address, all you will see for every client will be the SNIP or MIP address you are using. Here are a few examples I have personally had to deal with:

- IIS requires client IP for IIS logs
- IIS requires client IP for ACL filtering
- Web application requires client IP for it’s own logging purposes
- Web application requires cilent IP for authentication

Now to get around this for IIS, you can install ISAPI filters and set your Netscaler to use a custom header to store the true client IP address and pass that along with every packet. The ISAPI filter in IIS would look for this particular header and log that instead of the SNIP/MIP. There’s a very nice Client IP extraction ISAPI filter with some great instructions located on this page from Citrix or you can even write your own:

http://support.citrix.com/article/CTX119347

This is great for logging but sometimes your web application may grab the IP address from web server variables. To return the IP address of the host making the request, the server variable that your ASP, ASP.NET, and PHP code will call is “REMOTE_ADDR”. Unfortunately REMOTE_ADR is going to have your SNIP IP and not the client IP when passing through the Netscaler. The good news is that if you have control of your code, you can make it call from a different location that includes the client IP your ISAPI filter is passing. So if your code currently uses:

Request.ServerVariables("REMOTE_ADDR")

then switch it to:

Request.ServerVariables("http_client_ip")

where “http_client_ip” is the real client IP address and not the SNIP. Everything will work fine after that.

Here’s a nice snippet of code you can run on your web server and hit from your client browser to see what all the Netscaler is passing to the server:

<html/>
<body/>
<p/>
NS SNIP or MIP IP address:
<%Response.Write(Request.ServerVariables("remote_addr"))%>
<br/>
<b/>The NS Inserted Client IP:</b/>
<%Response.Write(Request.ServerVariables("http_Client_ip"))%>
<br/>
Accept Encoding:
<%Response.Write(Request.ServerVariables("HTTP_ACCEPT_ENCODING"))%>
<br/>
Cookies:
<%Response.Write(Request.ServerVariables("HTTP_COOKIE"))%>
<br/>
</body/>
</html/>

(***Note, I apologize in advance if WordPress messes up the code block above. It usually changes the quotes in the code around so if you have issues with it, I will host a .txt file here with the code. Just comment to this post if you need it.)

This is all well and good when you have access to the code and can make this change but what about web apps that are 3rd party and you cannot modify the code? Well then you have no choice but to use the USIP (Use Source IP) feature on the Netscaler. What this does is pass the client IP straight through the Netscaler to your backend server so that the SNIP or MIP you are using on your Netscaler is never seen by the server (except for monitor probes from the Netscaler itself of course). This option is disabled by default. It can be applied to the entire Netscaler or turned on and off per service.

Best practice is not to use it and avoid it as much as possible. The reason is that using the USIP feature means you are going to lose very important features on the Netscaler such as connection multiplexing and surge protection. It’s always better to use the Client IP header insertion instead. In fact to date I have personally never enabled USIP in any environment I have built so far.

Here’s a very excellent and recent article from Citrix on the pros and cons of using USIP mode:

http://support.citrix.com/article/CTX121974

I hope this helps anyone trying to get around Client IP address related issues. Please feel free to post a comment on your own experiences or suggestions.

The GUI on a Citrix Netscaler is nice but it takes 7 seconds to update and sometimes you need live stats and a 7 second delay is not going to work for you. You can do A LOT through command line you can’t through the GUI (isn’t that the case with pretty much anything though? ). I am writing this article to show you how to get stats on each of your VIPS and servers quickly and also monitor live connections and traffic into your appliance.

 

 

GETTING DETAILS
First, you might want to get some details on your Load Balanced VIP. I am calling my Load Balanced VIP “lb_vsvr_name” in this example. Here is how you do that:

1. SSH into the Netscaler using Putty
2. Type:

sh lb vserver lb_vsvr_name

You will immediately see a lot of stats on how this VIP is configured like it’s IP, it’s status, connection method, persistency, bound service groups, etc. Pay attention to the bound service groups. So to get details on one of those, type this where “svcgrp_name” is the name of your service group:

3. Now type:

sh servicegroup svcgrp_name

This will show you details on the service group such as the status and the individual back end servers. Let’s say you have two servers called “svr_name1″ and “svr_name2″. To get the details on those backend servers:

4. Type:

sh server svr_name1

and

sh server svr_name2

This will pull up stats on each of your backend servers.

 

GETTING REALTIME GENERAL PERFORMANCE STATS
1. Make sure you are still SSHed into the Netscaler using Putty
2. Type “shell” to enter the FreeBSD shell
3. Capitalization matters below or you will get an error. Type:

nsconmsg -s ConLB=2 -d oldconmsg

What this does is pull up realtime general stats on all your VIPs and services and continues to update it every 7 seconds until your press Ctrl+C to stop it. I know the display has a ton of info in it but Citrix actually has some very good documentation on how to read it all here: http://support.citrix.com/article/CTX113341

 

GETTING REALTIME TRAFFIC & CONNECTION STATS
Now that you have verified the VIP is up and have it’s IP and bound service groups. Here is how you can SSH into the Netscaler and get live traffic and connection stats for the VIP itself:

1. Make sure you are still SSHed into the Netscaler using Putty
2. Make sure you are not in the shell. Type “exit” if you are still in it from doing the previous section.
3. Now type:

shconnectiontable

This will give you every connection coming into the Netscaler. The columns displayed are in this order:

SRCIP = source IP address
SRCPORT = source port
DSTIP = destination IP address
DSTPORT = destination port
SVCTYPE = service type like HTTP, SSL, etc.
IDLTIME = idle time of that connection in milliseconds
STATE = current state of the connection, you will usually see it either waiting or established

Pretty hard to look at if the device you are looking at is in production and has a ton of traffic on it. So we need to filter this display. Let’s just look at how many connections we have to just one of the backend servers above. Type:

sh connectiontable | grep xxx.xxx.xxx.xx

or

sh connectionable "ip == xxx.xxx.xxx.xx"

where xxx.xxx.xxx.xxx is the IP address of one of the backend servers above. This will give you stats on all the connections to that backend server.

1. Anything that says “MONITOR” is the Netscaler itself testing for uptime so ignore those.
2. If you see “MONITORS”, ignore those too, they are just SSL based monitors.
3. If you see “HTTP” or “SSL”, that’s the real traffic you are after. Note if they say “TIME_WAIT” or “ESTABLISHED”. If it’s live traffic, the state should be an established connection.

Let’s get into some more advanced filtering. Let’s say you want to filter all monitors to that IP and only display SSL, HTTP, etc:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR"

Let’s say you want to see all traffic to that IP but only established connections. Here is the syntax for that:

sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established"

Let’s say I want all the traffic to that IP, established connections only, and only display SSL connections:

sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established && svctype == SSL"

 

One thing to note, typically you can filter additional service types like this:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR && svctype != SSL"

but I found that if you have “MONITORS” and you wanted to filter them, it does not work and gives you an error saying it’s an invalid command. I actually have an open ticket with Citrix about this bug so it should be fixed in a future release. In the meantime, just use a combination of a filter expression and grep to filter it out. Here’s an example:

sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR" | grep -v MONITORS

 

FURTHER READING
If you need further info on commands and syntax available on the Netscaler, I have hosted the Citrix Netscaler Command Reference Guide for 9.1 here:

NS-CommandReference-Guide.pdf

Yes it’s 1742 pages but take the time to flip through it. It’s extremely powerful. The connectiontable related syntax begins on page 883 (873 in the guide).