Mitigating DDoS and brute force attacks against a Citrix Netscaler Access Gateway

July 2nd, 2014 No comments

DDoS and brute force attacks are constantly evolving and becoming more widespread. It’s not just high profile websites like financial institutions and e-commerce sites that get attacked anymore. Any organization can be a target in a matter of minutes. A few sample scenarios could be there’s a press release about a new product your company is releasing so they use a DDoS smoke screen to gain entry into your network to cover data theft. Or your whole industry is getting the attention of the media and you’re one of the top companies in that industry so whichever groups takes you down will be in the spotlight. Or your CEO does something notable and is in the news and so anything associated with him will be in the news including an attack against his company. Or unfavorable legislation passes for your industry and public opinion is low, ripe pickings for hacktivists. Or the attacker just plain felt like attacking an entire group of sites to test out a botnet and yours just happened to be one of them. The possibilities are endless so don’t think you will never be targeted because nothing has happened in the past. Check out http://www.digitalattackmap.com/ for a live view of global DDoS traffic. You should be able to confidently say you have a mitigation plan to keep one of these attacks at bay.

There are several things you can do to protect your Citrix Netscaler Gateway (Access Gateway) from DDoS/DoS and brute force attacks. First off, DDoS protection should be in front of the Netscaler in my opinion. In front of the firewall even. Look into DDoS protection from your ISP if they offer it or an onsite solution that sits in front of the firewall like appliances from Corero or Arbor Networks. You can even protect the ISP router if you like with some of these solutions. There are also DDoS mitigation/scrubbing services like Prolexic (now part of Akamai) or CloudFlare you can look into.

It’s not just traditional SYN flood attacks at the network level you have to worry about anymore. There are reflection\amplification attacks utilizing DNS, NTP, etc. There are fragmented packet attacks causing your appliances to wait around needlessly for missing packets, there are application layer attacks against your web server or app itself, and there are the worst kind which is specialty crafted packet attacks against the appliance or back end server OS itself. There’s always something new around the corner which is why you really need a dedicated on-site appliance or external service to help analyze and mitigate any and all attacks including 0 day attacks. You want to be pro-actively alerted to and remain vigilant against attacks, not scrambling during and after the attack.

HOW YOUR CITRIX NETSCALER CAN HELP
Even if you implement all these layers in front of your Netscaler, there are a few things you should configure on the Netscaler itself for an extra layer of security. Netscalers are very powerful devices can can help defend against regular DoS and to some degree DDoS attacks very well. You’ll need a Platinum level device for the full set of security features such as DoS shield, Day Zero attack prevention/Application Firewall, etc. You have the capability to do the following:

1. Application layer defense (Layer 7 defense) – like Slow Read, Slow HTTP POST (RUDY), Slow HTTP Headers (slowloris), HTTP Flood, Random Searches, Apache Range Header, etc. that cause resource starvation on the web servers (not usually a volumetric attack)

  • Application protocol validation (enforcing RFCs thus dropping malformed packets and the ability to create your own validation using Content Filtering or Responder policies)
  • Surge protection/priority queuing (presents traffic to backend servers based on their current capacity while rest gets queued until resources free up. priority queuing can weight certain kinds of requests providing them preference in processing)
  • HTTP flood protection (HTTP GETs are challenged)
  • HTTP low-bandwidth attack protection (monitoring for excess slow connections and proactively reaping)

2. Transport layer defense (Layer 4 defense) – like SYN floods, DNS query floods, SMURF, SSL floods, etc. to starve bandwidth on the web servers by using up sockets (volumetric attack)

  • Full-proxy architecture (nothing in without an explicit policy you set)
  • High performance design (low latency processing, purpose built HTTP parsing engine, hardware accelerators)
  • Intelligent memory handling (memory isn’t allocated until connection is validated)
  • Extensive DNS protection (running either DNS proxy or authoritative modes protects you with DNS specific validation, rate limiting, etc).

3. Network layer defense (Layer 3 defense) – DNS amplification, IMCP floods, UDP floods, teardrop, fraggle, christmas tree, etc. to starve bandwidth on network devices (volumetric attack)

  • Embedded defenses (SYN cookies)
  • Default deny security model (dropping invalid packets that are not defined in your policies)
  • Protocol validation, rate limiting (can be done using AppExpert/Responder policies)

Once you have your stack of DoS/DDoS protection in place, test it. During a maintenance window, attack your own network (please let your Network/Security/IT Management teams, etc. know what you’re doing ahead of time). See how your defenses behave. There’s no point in putting up a giant expensive wall against a flood if it has cracks in it. See what your limits are. Download LOIC, Slowloris, Hping3, HOIC, Kali, etc. and let them loose. If you don’t want to do it yourself, there are tons of vulnerability and penetration testing firms out there that can help you. Make sure there are members of your team that are getting IT-ISAC, FS-ISAC, US-CERT, etc. notifications and actively engaging peers in your business sector to see what they have come across. Ensure your organization has an incident response plan specifically for DDoS attacks.

WHAT YOU SHOULD CONFIGURE ON YOUR CITRIX NETSCALER
Now let’s get into the technical stuff. So how do you enable defenses on the Citrix Netscaler? Specifically around the Netscaler Gateway (Access Gateway)?

1. Configure Max Login Attempts & Failed Login Timeout – This can be done globally on the Netscaler Gateway (Access Gateway) but I prefer to do it on the Netscaler Gateway vServer itself. Go to Netscaler Gateway > Virtual Servers > double click your virtual server, and select the number of attempts and timeout. You should set the number of attempts to just 1 attempt below what your AD account lockout policy is set to so that user accounts are protected from being locked out in AD, let the Netscaler do the lockout for you. In this example I have 5 attempts and a 15 minute timeout set in a scenario where my AD account lockout policy is set to 6 failed login attempts:

1

If a user calls the help desk frantically saying they need to login and the help desk sees their account is not locked out but resets their AD password, they will still need to wait 15 min before attempting to login again on the Netscaler Gateway because it is still locked out there. There was no way for you to be able to “clear” their user ID for them from the Netscaler GUI. I’ve been asking for this feature for sometime and with the 10.5 firmware release yesterday, this is may now be possible (I haven’t had a chance to test it yet though): http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-5-map/netscaler-10-5-rn.html#netscaler-10-5-rn__netscaler-10-5-50-x-mr-enhs

Also note, this is all keyed off the user ID, not client IP. If you type “jsmith” for the user ID and try 5 different incorrect passwords, you will get the max login attempts message. But if you try “jsmith”, “jdoe”, “jlee”, etc. it won’t trigger the max login attempts. Also if they just type a user ID and no password, it won’t trigger a login attempt either. I would like to see this changed where any click of the “Log On” button is considered a logon attempt instead and have submitted a feature request for this. My concern is that if an attacker has a list of user IDs (easily guessed or extrapolated), then can sit there and nail the Netscaler Gateway attempting to brute force the password causing an Active Directory account lock out. The help desk gets a call from the user, they unlock the account, and immediately it gets locked again because of the attacker. How’s the user supposed to work if their account keeps getting locked out? As a work around, you can create a responder policy that looks for the POST from the Logon button being clicked to mitigate this threat. Look for a POST going to https://yourdomain.com/cgi/login to help build your policy.

2. Rate limiting – Prevents an attacker from brute forcing the login page. If they hammer the page too many times, the Netscaler will drop the packet. In this example I’m going to configure a Responder policy to drop packets if they exceed more that 4 page requests in 10 seconds.

-Go to AppExpert > Rate Limiting > Selectors and create a new selector with these two expressions:

CLIENT.IP.SRC

HTTP.REQ.URL

so your Selector should look like this with them stacked on top of each other:
2

-Now create a Limit Identifier that references the Selector. For the sake of testing, I have set it to 4 requests within a 10 second period. Please don’t use this in production because the time slice is way too high. This is just so we can test easily:

3

-Now create a Responder Action if you intend to redirect the user. In this example I want to drop traffic so you don’t need a Responder Action for this. It can be done on the Responder Policy itself. So go ahead and create a Responder Policy like this with an Action of DROP and the Expression to check for the Limit Selector you created before:

4

-If you do syslogging (I hope you are – see http://www.jasonsamuel.com/2014/05/28/best-practices-for-citrix-netscaler-aaa-logging-and-retention/), you can create a log action here so you can be notified in the event of a brute force or DoS attack. Just create a new Log Action like this on your Responder Policy. I like to log it to the newnslog as well. You will need to bypass the safety check to get it to apply or it will warn you it’s unsafe:

5b

You’ll also need to go to System > Auditing > Settings > Change global auditing settings > check the “User Configurable Log Messages” box for both SYSLOG and NSLOG auditing types. Otherwise you’ll see hits on your Responder policy but no hits on your Auditing Message Action policy:

user-configurable-log-messages-netscaler-syslog-nslog

Once done, refresh your Netscaler Gateway login page time 5 times. On the 5th time it should start dropping. Use the HTTPFox FireFox add-on to watch it if you like. You should also be able to go to your Responder Policy and watch the hit count rise. Don’t forget to adjust your threshold and time slice to something more realistic after your testing is done.

You can view your Responder policy hits from the CLI by using:

nsconmsg -K /var/nslog/newnslog -d current | egrep -i responder

and Message Action hits by using:

nsconmsg -K /var/nslog/newnslog -d logfromnfw

or

show audit messageaction

and to view the actual message:

show audit messages -loglevel ALERT

If you chose to not write to the newnslog and are instead writing to the ns.log, you can use:

tail -f /var/log/ns.log

Just don’t forget about the 7 second delay before the hits appear in your Putty window.

One thing to note, remember the AOL days in the 90′s? Every user was behind the same proxied IP address. Blocking an IP would block pretty much all AOL users. This changed in 2006 but even Wikipedia had trouble blocking malicious AOL users before then: https://en.wikipedia.org/wiki/Wikipedia:AOL. If you do business with another large organization and they are coming through a proxied IP, they could potentially trigger the rate limiting if the time slice is too large and several users tried to login at once. You would unintentionally block legitimate traffic. So keep an eye on this. The syslog email alerts will help you out big time to detect this and make adjustments as necessary before your service is impacted.

3. TCP optimizations – I’ve Tweeted about this before. You should be using the “nstcp_default_XA_XD_profile” on your Netscaler Gateway (Access Gateway) virtual server. This has optimizations in it designed around XenApp and XenDesktop. ICA traffic is a lot of tiny packets so a lot of back and forth communication. This TCP profile speeds up ICA transmission by cramming more packets into each transmission. There are 3 very good blog articles from Citrix by Abhilash Verma covering tuning the TCP stack using profiles:

http://blogs.citrix.com/2012/03/22/tune-netscaler-tcp-stack-to-suit-your-needs/

http://blogs.citrix.com/2012/03/29/create-your-own-tcp-optimization-profile-on-netscaler/

http://blogs.citrix.com/2012/08/02/get-best-tcp-performance-for-xenapp-xendesktop-deployments/

Now the “Nstcp_default_tcp_lnp” profile is recommended in low bandwidth scenarios so I’m not sure if during a DDoS attack it is best to switch the profile on your vserver temporarily to help keep your ICA traffic flowing. This is something I need to test.

7

6

Compare the screenshot between the default TCP profile vs the XA/XD specific one. To break it down:

  • TCP window scaling – increases the receive window size by a factor of 4. This is basically telling the client that the Netscaler Gateway can take on more packets than a typical standalone web server could so “bring it on”. For most scenarios where the client has plenty of bandwidth, this allows for a much better ICA session because it is sending more packets per transmission.
  • MSS – Maximum segment size is exactly what it sounds like, the largest amount of data you can send in one TCP segment. It goes from the default 1460 bytes to 0 bytes. The bigger the segment, the more fragmentation can occur as it travels over all the Internet routers/hops which causes the connection to slow down (latency). The lower the number, the less fragmentation but slightly more overhead on the Netscaler. By dropping the MSS to 0 bytes, I BELIEVE the Netscaler is telling the client it can go as small as the client can can go to prevent fragmentation. But on other network devices I’ve seen an MSS of 0 mean disable so I’m not 100% sure about the setting. I’m going to trust the Netscaler product team’s judgement on this one. :)
  • Max packets per MSS – Bumped from 0 to 10.
  • Max packets per Retransmission – Bumped from 1 to 3. This is saying send 3 packets instead of 1 per retransmission.
  • Minimum RTO – Dropped from 1000 milliseconds to 200 milliseconds. This speeds up retransmission rate drastically.
  • Selective Acknowledgement – This is now checked. SACK allows the receiver to acknowledge out of order blocks of packets which were received correctly and the last packet that was received in the right order. This prevents the sender from having to retransmit data he thought was lost at a slower rate. The sender gets the SACK and just continues transmitting away at high speed. It only has to retransmit packets that were truly lost.
  • Nagle’s algorithm – This is now checked. This reduces the number of packets required to be sent. ICA is a very chatty protocol so tons of small packets. The algorithm buffers small packets and sends them as a group instead of one at a time to increase transmission efficiency.

Now compare that against the lnp profile:

8

4. Alerting – Use AAA syslogging to alert you of suspicious login activity. Splunk, Kiwi, Command Center, etc. Set an alert that if you see X number of invalid logins within a couple of minutes period, send an email. See http://www.jasonsamuel.com/2014/05/28/best-practices-for-citrix-netscaler-aaa-logging-and-retention/

5. External monitoring – Always setup external monitoring of your URLs from multiple data centers around the world. Pingdom, Site24x7, Dotcom-Monitor, etc. are all good services for this and even offer some degree of free monitoring. I prefer extended validation when setting up monitors, not just looking at HTTP response codes or headers. Look for something near the footer of the web page like a copyright line. Until that page is fully loaded, I wouldn’t consider it a successful poll. Make sure to set response time alerts. If a page that normally takes 1 second to fully load is all of a sudden taking 10 seconds, you have a problem and need to know about it.

6. Don’t rely completely on IPS/IDS – Intrusion Prevention Systems and Intrusion Detection Systems are a good layer of security but that’s it, it’s just a layer. Typical HTTP floods will look like legitimate traffic to the IPS. And it will sit there and track each and every one of those connections quickly exhausting resources. An IPS is not designed with DDoS mitigation in mind where there could be tons of legitimate looking traffic thrown at it. It’s designed to inspect individual traffic streams and analyze behavior.

7. Obfuscation – I wrote about using your Netscaler to perform server header obfuscation several years ago to help prevent HTTP fingerprinting: http://www.jasonsamuel.com/2011/04/18/using-your-netscaler-for-http-server-header-obfuscation/. It’s an easy rewrite policy to configure and adds just one more layer of security. Now anyone that works on Netscalers on a daily basis can spot a Netscaler Gateway page a mile away no matter what skin or URL rewrites are being done so don’t rely on this extensively. It’s just an extra layer of security to slow down an attacker and I usually use this on my load balanced vservers more effectively which are usually custom websites than on Netscaler Gateway (Access Gateway) vservers which is pretty common. The article I reference in that post from Citrix is no longer available but here’s a quick example of how I do it:

-Create an Rewrite Action with the Rewrite Action of REPLACE
-Set the expression to:

HTTP.RES.HEADER("Server")

-Set the replacement string to:

"Server"

(in my example I set it to “jasonsamuel” for you easily see the change)
1

-Now create a Rewrite Policy that binds to this action. Undefined Action is:

NOREWRITE

and Expression should be:

HTTP.RES.IS_VALID
2

then go ahead and bind this to your Netscaler Gateway vserver. Make sure to choose Rewrite (Response) and not Rewrite (Request) or it won’t bind.
3

You can also set this globally via GUI. Just go to System > Settings > Configure HTTP Parameters > Server Header Insertion. Check it to enable it and type in whatever you like in the box to set the server header. I never do it this way and prefer to do it via the rewrite policy and bind globally.

Anyhow, use the HTTPFox or Live HTTP Headers FireFox addons to quickly verify the change:

BEFORE
before

vs.

AFTER
after

8. Verify management IPs – The Netscaler by default is set to only allow administration on the NSIP for management tasks. But if you walk into a new environment where someone else stood up the Netscaler, they could have enabled it on a SNIP, MIP, or VIP for whatever reason and this is a huge security issue if an attacker uncovered it. Always double check.

9. Verify all local accounts – Audit them. Document them. Change the default nsroot password to a 25 character plus passphrase. Use integrated AD authentication if possible and set granular policies for any admins that don’t need full access to all modules on the Netscaler.

10. Do not enable verbose login error messageshttp://blogs.citrix.com/2014/06/11/enhanced-authentication-feedback/ talks about enabling Enhanced Authentication Feedback. Don’t do it. Never give an attacker any information to work with. If you’re troubleshooting something, use AAA logging instead.

11. Kill all weak SSL ciphers – I talked about this a few years back: http://www.jasonsamuel.com/2010/08/09/verifying-that-sslv2-and-weak-ciphers-are-disabled-for-pci-compliance/. Verify your are allowing only the strong SSL/TLS protocols and weak SSL ciphers are disabled. You only want to enable what you need to pass whatever compliance your organization may have to meet (SOX, PCI-DSS, HIPAA, FERPA, etc.)

12. 2 factor authentication – RSA SecurID, Symantec VIP, etc. are all supported. Tons of SMS based services too if you don’t want to use soft tokens (smartphone apps) or hard tokens (keychain fobs). Remember, the Netscaler Gateway can do primary and secondary authentication. Set the one time password (2 factor auth token) to be the first and set the AD authentication policies to be the second. That way valid AD accounts are protected from lockouts by an attacker. There is no excuse to not use 2 factor authentication in your organization. It is dirt cheap to implement and mitigates so much risk you’d be stupid not to. Use it.

13. Setup Command Center and Insight Center – They’re both free so use them. Depending on your license level you’ll get a certain feature set. If you’ve got a Netscaler, there is no excuse. You need to be running them. The automated config backups of Command Center alone can save you in the event of a catastrophic attack. Insight Center can help uncover unusual behavior and traffic spikes:

Citrix Command Center:
http://support.citrix.com/proddocs/topic/netscaler/cc-gen-commmand-center-wrapper-con.html

Citrix Insight Center:
http://support.citrix.com/proddocs/topic/netscaler/ni-main-wrapper-con.html

14. Managed IPS/IDS & Firewall log review – There are so many managed services out there to review your firewall and IPS/IDS logs in realtime. There are companies with NOCs dedicated to respond to issues 24/7 so you don’t have to. Look into something like this from Alert Logic, Trustwave, SecureWorks, 403 Labs, etc. if you don’t have the resources available on staff. I’ve been in a couple of situations in my career where a phone call from one of these third party solutions uncovered something internal monitoring missed and we were able to respond to the issue before it became a bigger problem. As I said above, make sure you enable syslogging and see if one of these companies can watch for Netscaler related activity for you. Even a spike in CPU you could easily see using SNMP monitoring could indicate a problem and having someone dedicated to monitoring will come in handy.

I hope this article helps you out. Please do leave a comment below if you have any DDoS or brute force attack mitigation techniques you’d like to share.


Firefox 30.0 users not able to launch Citrix XenApp or XenDesktop sessions

July 1st, 2014 No comments

I’ve received reports some Firefox users are not able to launch ICA sessions (both XenApp & XenDesktop) after the latest Firefox 30 update. Firefox 30.0 sets all Plugins to “Ask to Activate” as a security feature including Citrix Receiver (Citrix ICA Client 14.1.0.0 in this example):

1

The symptoms are when a user clicks a XenApp application or XenDesktop VM, you get the grey circle as if it’s about to launch and then nothing. I’ve tested on both Web Interface 5.4 and StoreFront 2.5 with the same result. It’s definitely a plugin permission issue.

2

If you look to the left of the address bar, there is a little grey block you can click on. Once you click on that, it will ask if it should allow running the “Citrix ICA Client”:

3

Just hit “Allow” or “Allow and Remember” and try launching again. The app or VM will launch now.

Though this is considered to be a security feature, Mozilla is tracking this as a bug here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1025627

and there are a few posts on the Mozilla Support forums about this:

https://support.mozilla.org/en-US/questions/1005678

https://support.mozilla.org/en-US/questions/1005650

Mozilla’s official stance about disabling plugins by default: https://blog.mozilla.org/security/2014/02/28/update-on-plugin-activation/

It looks like Citrix and other organizations can have their plugins “whitelisted” here: https://wiki.mozilla.org/Plugins/Firefox_Whitelist

Here is the current list of plugins as of July 1st, 2014 (surprisingly not a whole lot):

Whitelist Request Plugin Name Vendor Start Release Application Expires
979849 Unity Player Unity 30 34
980133 Jabber SDK Cisco 30 34
980772 McAfee Security Scanner McAfee 30 34
981403 VGConnect for DirecTV Cisco 30 34
981905 Jabber Guest Cisco 30 34
982045 Estonian ID Card RIA 30 34
984441 coupons.com Plugin Printer coupons.com 30 34
987056 Nexus Personal BankID Finansiell ID-Teknik BID 30 34
988119 GradeCam GradeCam 30 34
988781 Smart Card Plugin Adrian Castillo 30 34
989096 WebEx Cisco 30 34
990067 Skype Microsoft 30 34
990068 Facebook Video Calling Microsoft 30 34
990069 MS Office Lync Microsoft 30 34
990286 VidyoWeb Vidyo 30 34
981503 McAfee Virtual Technician McAfee 30 34
989872 ViewRightWeb Verimatrix 30 34
987057 SiteAdvisor Enterprise McAfee 30 34
985640 F5 Networks SSLVPN/Firepass F5 Networks 30 34

I’ve reached out to a few of my contacts at Citrix about this so hopefully the plugin is in the process of being whitelisted. Otherwise there’s going to be a whole lot of Firefox users calling your help desk. I’ll update this post as soon as I hear something.


Best practices for Citrix Netscaler AAA logging and retention

May 28th, 2014 No comments

By default the Netscaler is set to certain log levels for certain modules on the device, including AAA (authentication, authorization and accounting) logging. Sometimes you may want to change the AAA log retention temporarily for easier troubleshooting.

I’ve posted several articles around Netscaler AAA already but if you’re new to it, AAA logging is saved to the /var/log/ns.log file (and all subsequent rollover files i.e ns.log.0.gz, ns.log.1.gz, ns.log.2.gz, etc.) and you can view these via GUI by going to System > Auditing > then clicking on “Syslog messages”. Once the window loads, in the Module drop down choose “AAA” or whatever you are trying to filter for and hit “Find Now”. Choose the logs in the bottom left (just doubleclick).

ns0

The default setting for AAA logs is set to save the last 25 log files (circular logging where it will overwrite the oldest) and the size is set to 100 Kilobytes. Not a whole lot. The best practice is not to utilize the device itself for historical logs. It’s really only meant to be used for realtime/neartime logs for troubleshooting purposes. Historical Netscaler AAA logs should be offloaded from the appliance using a syslog server. It’s very easy to configure. Just go to Auditing > Syslog and add your syslog server and policy. Make sure you select “ALL” events. It’s always best to have more and let your syslog server do it’s thing than try and filter on the device and end up with something you needed missing. Forward it all. Your syslog server should be sized to take it and retain the data for whatever period your information security/compliance policy dictates.

If you don’t have a syslog server available, there’s no excuse. Stand up a free edition of Solarwinds Kiwi or Splunk: http://www.solarwinds.com/products/freetools/free-kiwi-syslog-server.aspx or http://www.splunk.com/view/SP-CAAAE8W. If you have multiple Netscalers you might already be running Citrix Command Center (which is also free) and has syslog ability: http://download.citrix.com/downloads/command-center.html.

Now, to actually view the settings for each log, WinSCP in and go to:

/etc

and copy over the newsyslog.conf. If you open it in Notepad it’s kind of hard to make out. Only use WinSCP to make a backup. Don’t try and edit it with Notepad and push it back.

To view and edit this file easily, use Putty instead and SSH in. Then go to:

1. shell
2. cd /etc
3. cat newsyslog.conf

ns1

Right along the top, there is a little warning. Any changes to this file are lost after rebooting. Whatever changes you make, you need to copy into the /nsconfig folder. By default there isn’t one in that folder until you copy it over.

This will show you the contents of the configuration file. See the line that says “/var/log/ns.log”? Those numbers to right are the settings for the ns.log. As I mentioned before, by default is set to save the last 25 log files and the size is set 100 Kilobytes. That’s not a whole lot.

4. Now before you decide to increase the size or number of logs files that are kept on the appliance, check to see how much disk space you have available. Type

df -h

What you’re interested is in the /var partition which you can see on my test device has plenty of room left in my screenshot.

ns2

5. Now let’s edit the newsyslog.conf. I don’t want to go overboard here. In this example I’ve calculated that if I bump the log from 100 K to 3 MB I should be okay. That’s 3 MB x 25 logs = 75 MB. That’s a little bit of extra logging I need for troubleshooting users who can’t login with out having to pull logs from my syslog server. Again, don’t max it out because the appliance itself is not where you want to be storing logs. To edit, type:

vi newsyslog.conf

and arrow your mouse down to the 100 and type “i” to edit/insert. Now type 3000 to make the size 3 MB. Once done, press the ESC key and type:

:x!

which will save and exit. If you screwed up and want to exit without saving your changes, type:

:q!

ns3

6. After you’ve made your changes, you need to copy the file to /nsconfig so it persists after device reboots.

7. Finally, you may need to restart the syslog process for the changes to go into effect (or just reboot the device if you have a maintenance window). In my experience I usually don’t have to but it’s always best to test to see if your changes persist through a reboot. Type the following to find the syslog process ID:

cat /var/run/syslog.pid

then to kill the process type:

kill -9 <TheProcessIDNumber>

and finally to restart the process:

/usr/sbin/syslogd -b 127.0.0.1 -n -v -v &

This is all FreeBSD so if you’re no familiar with the switches:

-b will bind the syslog to the localhost
-n run in the foreground (as in wait until the process completes)
-v -v (yes that’s two of them) maxes out the verbosity of the logged messages

You can see below in my screenshot my process ID changes after the process is restarted. This is a good way to verify the process actually restarted:

ns5

8. Go back to the GUI syslog and check the latest log (hopefully it has turned over by now). You’ll notice the first entry in the logs will begin say:

logfile turned over due to size>3000k

or whatever size you had specified.

ns4


Upgrading from XenDesktop 5.6 VDA to XenDesktop/XenApp 7.5 VDA

May 23rd, 2014 1 comment

You may run into a little bit of a challenge upgrading a Windows 7 system from the XenDesktop 5.6 VDA (I was using 5.6.200.9 on this particular machine) to the latest XenDesktop/XenApp 7.5 VDA (7.5.0.4523). If you uninstall the 5.6 VDA, reboot, then try to install the 7.5 VDA expecting a clean install, it will fail and you’ll get the always wonderful and completely generic “InstallFailure 1603″ error message:

vda install failure

When you View error details, you’ll see something like this:

Error Id: XDMI:CFF2362A

Exception:
Citrix.MetaInstaller.MetaInstallerException Installation of MSI File ‘BrokerAgent_x64.msi’ failed with code ‘InstallFailure’ (1603).
at Citrix.MetaInstaller.Msi.InstallProduct(InstallationContext context, String msiPath, String parameters)
at Citrix.MetaInstaller.MsiComponent.Install(InstallationContext context)
at Citrix.MetaInstaller.InstallationManager.InstallComponent(IInstallableComponent component, InstallationContext installContext)

Apparently this is an issue related to .NET changes by Microsoft. Even if you uninstall and re-install each .NET version manually, the install will fail. I’m aware of at least 4 incidents lodged with Citrix Support on this issue since 7.5 was released so not everyone may be impacted. It’s been kicked up to development and I’m sure a fix is forthcoming.

In the meantime, it was suggested to re-image the system. That’s not going to fly with me. I tried several things to fix this and here is what finally worked for me:

1. Completely uninstall the “half-way” installed XenDesktop/XenApp 7.5 VDA
2. Uninstall any other Citrix components from previous VDAs if anything is left over (the only thing I left was Receiver)
3. Download and run CCleaner – http://www.piriform.com/ccleaner/download
4. Click Registry on the side and hit “Scan for Issues”. You’ll notice a ton of Citrix related stuff left in the registry. Click “Fix selected issues” then reboot.
2

5. Once it’s back up, do step 4 again and you might still notice a few things picked up by the scan. Fix and reboot yet again if you do see anything.
6. Once it’s back up, intstall the 7.5 VDA and it should complete installation and register with the Delivery Controller

Hope this helps you out. I expect the next VDA version to resolve this issue and hopefully this workaround will hold you over till then.


Citrix Netscaler vs F5 Big-IP LTM & APM – Which ADC should you choose for Citrix delivery?

April 24th, 2014 1 comment

citrix-vs-f5-jasonsamuel.com

Netscalers are my favorite Citrix product hands down. But I like to be well rounded and play with all technologies if I can. I keep getting asked by my peers on my opinions on Netscaler vs. F5 for Citrix delivery and I’m tired of saying the same things over and over again so that’s why I decided to write this article. Something I can just link them to from now on. :)

DISCLAIMER #1
I’m biased toward Netscaler. How biased you ask? My relationship with Netscaler goes back years and years (the red and black days). Now think back to Christmas time when you were a kid growing up. Did you ever shake your wrapped presents under your Christmas tree trying to see what they were? Did you recognize the shape of a box and already have an idea of what the gift was? It’s that Hot Wheels race track set you’ve been hinting to your parents about for months, right? Remember that excitement? You couldn’t wait to tear through that gift wrap and play with it come Christmas morning. This is exactly how I feel when I get emailed a tracking number and know my MPX appliances are being delivered at the datacenter that morning. I can spot the box an MPX appliance gets shipped in from 50 ft away. To the point where I can lift up the box and tell you approximately what series appliance is in there just from the weight. Opening it up and smelling the device and the out-gassing of the cardboard and packing foam is the same as the smell of gingerbread and eggnog for me. Getting an HA pair racked, configured, and blinking Primary and Secondary are my Christmas lights. Seeing my traffic go through them and watching my policy hit counters rise is like setting up that Hot Wheels race track and watching those little cars spin around and around all Christmas day. That’s how much I love deploying Netscalers. It’s like being a kid again for me. Netscalers are magical devices and I know how to make them do whatever I dream up. Now that you REALLY know how I feel…

DISCLAIMER #2
I’m not a network engineer. Sure I can configure Netscaler, F5, Cisco, Juniper, SonicWall, etc. appliances like nobodies business from my experiences earlier in my career but I don’t consider myself a network engineer. I’m going to have more of a virtualization and application\VDI delivery spin to my opinions because that’s what I do on a daily basis. I’m not a dedicated network engineer where that is my one and only function everyday by any means. So if you want a qualified purely network engineer perspective, you’ll have to find someone else to talk to. If you want the opinion of a Citrix Architect whose life revolves around XenApp and XenDesktop delivery, keep reading.

MY OPINION ON APPLICATION DELIVERY CONTROLLERS
F5 has been around forever but I never actually got to deploy them like I have over the years with Netscaler until last year when I played with LTM APM to see how well it handles Citrix XenApp and XenDesktop delivery. Netscaler and F5 are pretty much neck to neck for basic web traffic. All the Gartner, InfoWorld, etc. reports all show around X% of the market share belonging to one, and around X% to the other depending on whatever sources they’re using. Screw the marketing statistics. Get the answers yourself. If you have peers at Fortune 500s, and ask them what they’re using for regular web traffic. Then ask them what they are using for Citrix delivery or any Cloud based initiatives and why. You’ll find that depending on the use case, the ADC they are running will usually swing one way or the other almost every time and for good reason.

With Cisco getting rid of their ACE and partnering with Citrix to push Netscaler, many companies are left having to pick between Netscaler and F5 now. They are the market leaders. Even if you’re not intending to use them as ADCs (Application Delivery Controllers) and just plan on using them solely as a load balancing platform, you’ll find fans on both sides. If you haven’t had to pick yet, trust me you will soon because the ADC market is hot and your Cisco appliances aren’t getting any younger. Cloud and Mobility are keywords that just seem to get bigger and bigger as the months go by. I’ve been hit over the head with those keywords by every vendor on almost a weekly basis. It’s the first words out of their mouths even before “Hello” now. If you don’t have an initiative or project with these buzzwords in it, believe me, you will soon. And everyone wants a piece of your budget.

WHICH ADC SHOULD YOU CHOOSE?
So which appliance is better? Netscaler or LTM? That’s a hard question to answer and really depends on what you are trying to accomplish. Let’s just put the nitty gritty technical capabilities of the devices aside. You can find that info all over the place. Both are competitive and you can do almost all the same things. If not, wait a firmware version or two and you can. If you want to look at connection limits, throughput, and pricing, you can get all that info yourself from their websites and sales people. And there’s even literature each company will put out like this that highlight reasons to pick one over the other:

9 reasons why Citrix NetScaler beats F5:
http://www.citrix.com/content/dam/citrix/en_us/documents/products/reasonsnsf5.pdf

F5 Comparative Performance Report:
https://www.f5.com/pdf/reports/F5-comparative-performance-report-ADC-2013.pdf

There’s also online forums you can speak to other engineers on. Here’s a good thread on NetscalerKB, it’s a little dated but you can speak to other guys like me and get different opinions:

http://www.netscalerkb.com/netscaler-qa/netscaler-vs-f5/

and of course both companies have official online forums where you can post just about anything and get real world answers:

https://devcentral.f5.com/questions

http://discussions.citrix.com/forum/29-netscaler/

Bottom line you can get technical answers anywhere. So let’s get down to a few non-technical things people don’t think about that sales people aren’t going to tell you. Just some real world perspective from someone who has deployed Netscaler for Citrix delivery for years. Again, this is opinion based purely on my experience with them. Here are the questions you need to ask yourself if you’re considering either device:

1. How easy is it to deploy for the novice engineer?
2. How good is support?
3. How responsive is the company to feature requests?
4. Is what I am trying to do with Citrix application and desktop delivery supported and will it work with all my Citrix products?

How easy is it to deploy for the novice engineer?
F5 LTM with the APM module and XenApp/XenDesktop iApp template used for SSL ICA proxy is pretty nice. Just from a deployment standpoint, it’s more in plain English and easy to understand than setting up a Netscaler Gateway (Access Gateway). To an engineer that’s been in the game a long time, it doesn’t matter, you can knock out either. But to the new engineer, someone who isn’t as familiar with the platforms, configuration of a new appliance can be a daunting task. A lot of companies may have IT staff that have never dealt with an ADC or the intricacies of proper Citrix delivery and may be a bit lost. Having a template guide you in plain English is a plus. If you are one of these types of companies, you probably tend to lean toward whatever gets you up and running the quickest. Be cautious here. Read a few articles or YouTube videos on how to set each device up. Understand what it takes and if your company has the skill-set necessary on hand to configure either device. If not, do you have a partner that has someone on-staff skilled at either device? I don’t mean the guy that just set it up in his lab over the weekend and is coming out Monday to deploy it for you. You need to find the guy that’s been deploying your chosen device for years and can do what you’re wanting to do in his sleep.

How good is support?
Local support. This really depends on where you live and who you know. If you live in a major metropolitan area with several Fortune 500s around, you’re likely going to have local support resources who can help you in a moments notice. This is a very big plus if you’re new to configuring an ADC and need some guidance. I live in Houston, TX. Home of several Fortune 500 companies. One of the major hubs of the oil, gas, and energy sectors. If I need help, I will find it here from both vendors. When you’re trying to decide who to go with, find out the local support situation. This makes a world of difference if you don’t have anybody on staff that’s a guru on the appliance. You just want the device working as you expect and move on as fast as possible. Not making 3 phone calls and wait for an onsite visit to be scheduled through a sales person. You want to make 1 phone call and have a guy from the company knocking on your datacenter door and taking immediate ownership of the issue. So just figure out the onsite support situation for your city and let that weigh into your decision.

Remote support. If I call Citrix support with a Netscaler issue, I can speak to support and start working the problem within 10 minutes with a Platinum support agreement. Most times I get someone in Ft. Lauderdale, sometimes I get overflow and routed to someone in India . Sometimes I’ve gone around proper support channels and reached out to a couple of people in Dublin I know. No matter what, I always know someone is going to help me out or at least get the process rolling. When I get someone I know by name and have worked with in the past, I can breathe a sigh of relief because I know how good they are and my problem is going to get resolved.

The one and only time I called F5 support, I had to wait 24 hours for support to call me back because it wasn’t a critical outage or “potential” outage. So I wasn’t immediately routed to an F5 support engineer. That really didn’t sit right with me. These devices cost thousands of dollars. They’re a big investment for most companies. If I’m taking to time to call support, that means I’ve hit a wall and I’m going to be pretty frustrated to begin with. Don’t aggravate me. If I call in to ask how the weather is there, put me on the phone with someone immediately and tell me it’s sunny with a 20% chance of rain. Don’t blow me off and make me wait a day for an answer. I don’t want to know the weather a day later, I wanted it yesterday. What good is paying for 24×7 support if all I can do is lodge a ticket? I’m going to say I probably just had a bad experience and all support calls to F5 aren’t like that, especially since I was doing a POC at the time. But just do your research and call support a few times before making an investment with either company. Get a loaner device or download a virtual appliance and try calling them for help setting it up.

How responsive is the company to feature requests?
If you’re weighing Netscaler vs. F5 and you have some pull with the sales people, you can get a loaner device of each and play with it first in your environment. You can set it up and see if it does what you want it do. Long ago when Netscaler was newly acquired by Citrix, there were certain features I wanted to see but the device wasn’t capable of doing them at the time. I put in my feature requests through the proper channels and after several firmware versions, low and behold my feature would appear. None the less there was considerable wait and I often had to “rig” a solution as best as possible as a workaround until then. So if you’re trying to decide, ask your sales people for specific examples of what they’ve done for other companies in your business sector. This is really important because one day you’re going to want to do something neither device is going to be able to do and you want piece of mind that the vendor is going to do whatever it takes to make it work for you.

I like the fact that F5 has iApps and iRules communities to share templates. The community based sharing of templates reminds me of the Solarwinds Thwack community (http://thwack.solarwinds.com). As some of my long time readers know, I love Solarwinds. I’ve uploaded several NPM customizations and APM templates to Thwack myself. Easy access to other infrastructure professionals around the world and the sharing and collaboration of your work is essential part of making a product the best it can be. It’s a win-win for everyone. F5 really understands this concept and is embracing it. I love the heck out of the Netscaler Support forum (http://discussions.citrix.com/forum/29-netscaler/) and would love to see similar configuration and/or template sharing at the community level down the road.

Is what I am trying to do with Citrix application and desktop delivery supported and will it work with all my Citrix products?
How much money has your company invested in Citrix and can you really afford to make the wrong decision with your ADC and jeopardize that whole environment? This is probably the most important point of this article and I saved it for last because I wanted you to get in my head first and see where I’m coming from with all this. So far you’ve read a lot of my opinions and my past experience. Now let’s move on to some more factual information.

Citrix Netscalers are designed for Citrix delivery first, then everything else as opposed to other ADCs which are often built for traffic management first, then Citrix delivery. Netscalers have a lot of proprietary features that other ADCs cannot replicate. The larger and more complex your farms are, the more heartache you will have trying to deploy a 3rd party ADC solution. If you’ve been deploying Citrix products over the years, you know every year they are becoming more and more intertwined. They are designed to be deployed as a very cohesive product stack.

Let me give you some reasons why I will always choose Citrix Netscaler over any other ADC if I’m going to use it to deploy Citrix XenApp or XenDesktop:

    1. Integration with Insight Center
    2. Integration with Command Center
    3. Integration with Desktop Director (single pane of glass)
    4. Visibility into ICA channels
    5. Works with all Citrix farm versions (even EOL ones)
    6. MicroVPN (Really important if you’re considering XenMobile MAM and MDM)
    7. ICA protocol related TCP optimizations (look into the nstcp_default_XA_XD_profile TCP profile)
    8. Full support of the Citrix stack due to cross team collaboration between XenApp, XenDesktop (VDI), and Netscaler product development teams. If one is making an update to their product, the other teams are going to know about it and make changes in their product that will guarantee full compatibility going forward. Go Google a deployment guide for F5 LTM/APM on XenApp/XenDesktop 7.x architecture and tell me what you find.
    9. Citrix fully supports an environment with Netscaler. They WILL NOT support an environment with a 3rd party appliance doing ICA rewriting or any manipulation of the session launch or connection. You start trying to replace Web Interface and StoreFront with a Webtop and I guarantee you’re going to have bad time. This is an official stance from Citrix and stems from reliability and security: https://support.citrix.com/article/CTX131547

FINAL THOUGHTS
F5 is an excellent appliance, and I would consider it in a heartbeat for other things if we weren’t talking about Citrix delivery. But we are. And I’m going to go with the validated Netscaler based design that is 100% supported every time. I’m not going to rig it. I’m not going to make concessions. I will always recommend what makes sense. What works the best with the investment we’ve all already made with our Citrix environments.

If you’re dead set on F5 or a 3rd party ADC for Citrix delivery due to whatever constraints at your company and you don’t run a complex Citrix environment, I hope this article still helps you think of some of the non-technical aspects of choosing an ADC solution and encourage you to do research in these areas. There’s a lot more to think about than just the tech specs and marketing brochures.