Author Archive

Prepping a golden image with AppSense in a Citrix Provisioning Services (PVS) environment

July 10th, 2013 No comments

Here’s a little script I wrote to help me prep golden images in provisioned environments (pooled VMs) with AppSense DesktopNow. There is an excellent best practices guide on Citrix Provisioning Server and AppSense but I’m not sure it’s publicly available because I couldn’t find it on or I would link to it. I got it directly from an AppSense SE a while back. This script clears the CCA GUIDs for you as recommended in this best practices guide. If you don’t clear the GUIDs then all your standard images come up and check in with the AppSense Management Server with the same GUID which can lead to very weird behavior like missing data or settings. You can burn it right into an admin tools folder in your private golden images or run it remotely, whatever you prefer. When I’m ready to go from private to standard, I just run this bat file before shutting down. Works great for me.

:: AppSense/Citrix PVS private image CCA prep script
:: Created by Jason Samuel -

:: This will stop the CCA service
net stop "AppSense Client Communications Agent"

:: This will wipe the CCA agent GUID in registry for group id
reg add "HKEY_LOCAL_MACHINE\Software\AppSense Technologies\Communications Agent" /v "group id" /t reg_sz /d "" /f

:: This will wipe the CCA agent GUID in registry for machine id
reg add "HKEY_LOCAL_MACHINE\Software\AppSense Technologies\Communications Agent" /v "machine id" /t reg_sz /d "" /f


Once it runs check the registry and you’ll see the keys are now blank:

WordPress doesn’t seem to want to place nice today and the registry switches are running off the page. Just highlight the entire block of code above and paste into Notepad to see it all. Alternatively, here’s the script as a .txt file so you can see the whole thing or even rename it to .bat and use it as is:


Hope this helps. Let me know if you have any PVS + AppSense best practices questions or suggestions.

How to use Citrix Netscaler Insight Center to report on web, HDX, and Access Gateway (AGEE) traffic

June 17th, 2013 9 comments

I love Netscalers, they’re my favorite Citrix product. But one of the features I have always felt they are a bit lacking on was reporting. You might get requests from management asking how many users are hitting the Access Gateway (AGEE), when are they logging in, what is the session duration, etc. There is some pretty good data in the Netscaler dashboard but it is real time. Not historical. In the past, I have personally used the following for historical reporting on AGEE and any of my load balanced vservers:

1. 3rd party tool like Solarwinds utilizing via SNMP on the Netscaler
2. Enabling web logging on the Netscaler and extracting the data I need using LogParser
3. Using Citrix EdgeSight to report on clients (WI/StoreFront vs. PNAagent traffic) but hard to segment multiple traffic sources (especially when a XenDesktop session is launched and apps are launched through Citrix Receiver Enterprise/PNAgent via Start menu which will always appear as local traffic)
4. Using Netscaler Reporting Tool

And then there are some other 3rd party tools like Splunk for Netscaler which I haven’t used yet so can’t comment on. Regardless, the methods I have used in the past are nice but are very time consuming to get the metrics I need. I always wished Citrix would address this and when Citrix Command Center was released a few years ago, I thought a full blown management utility would include reporting but unfortunately it didn’t.

This all changes with Netscaler Insight (or Netscaler Insight Center as it is being called now). It’s a virtual appliance running on a XenServer that uses AppFlow on the Netscaler to pull metrics down and display it in a self contained web portal. And it’s supposed to even plugin with XenDesktop 7 Director in the future. If you have any Netscaler appliances in your environment, you NEED to download Insight Center right now and at least start getting familiar with it. Really, I could see it quickly become a necessary component for a successful Netscaler deployment. It does way more than just reporting, think of it as a single pane of glass into your Netscaler environment. As we all know, in IT visibility is key. From an engineer who needs metrics for working on an appliance to a CIO who needs to justify the cost of appliances and licenses, you need some sort of easy and reliable visibility of what’s actually going on. You can read all about the features here:

(Make sure to click on the little video box on the right side of that page. It does a pretty good job of giving a high level overview of Insight Center)

I started writing this article several months ago after playing with Netscaler Insight 1.0. There was barely any info about the product at the time. I started taking screenshots but decided not to publish this article because the Netscaler Insight 2.0 product was coming soon and was going to be a more polished product. Honestly, the 1.0 product was lacking many features. My biggest sticking point with it was that you could not view Access Gateway traffic. The eDoc for it clearly said it could but there was no way to do it. I actually had to call Citrix and escalate to the engineering team to find out this was going to be a feature of 2.0. So I decided to wait it out till the 2.0 release before publishing this article.

NetScaler Insight Center 2.0 Build 112.13 was released on May 21st and you can download the standalone virtual appliance image or the 90 MB upgrade pack from 1.0 to 2.0 if you’re an early adopter like me from here (you’ll need to login with your MyCitrix ID):

Again, I started writing this article using NetScaler Insight 1.0 Build 72.53 and even as is with the missing features, it’s been pretty nice for a 1.0 product. It is a 840 MB .xva virtual appliance based on FreeBSD just like a Netscaler VPX. I’m going to use my prior screenshots of the product to go over the setup portion and then dive right into the upgrade to Insight Center if you’re an early adopter.

1. I did this with the 1.0 virtual appliance but setting up 2.0 should be no different. Once you get the .xva imported into XenServer, it’s going to ask you for the IP, subnet, and gateway when you click the Console tab just like setting up a Netscaler VPX. Just get all this entered and save. One thing to note, once it is is imported it takes up 120 GB so make sure you have plenty of storage before attempting to import the virtual appliance.


2. Now open up a web browser and navigate to the IP you entered for the appliance. You will see the Insight login page. The default user ID and password is just nsroot/nsroot.


3. Once you login, you will get this Getting Started page.


4. Click Configuration and add the NSIP and admin credentials of one of your Netscalers. When it connects, it’s going to try and enable the AppFlow feature on the Netscaler so you must connect with a privileged account:


5. It immediately saw my 3 load balanced vservers and their states. In the View dropdown, you can select from Load Balancing or Content Switching vservers in the Netscaler Insight 1.0 product.


What was really confusing at the time was that the eDocs stated Access Gateway vservers were supported but I could not find any way to view it. Even the one and only video on Insight posted on CitrixTV clearly showed the 1.0 product with the exact same build I had showing AGEE hits:


It wasn’t until contacting Citrix, speaking to support who couldn’t answer it, and then finally getting a response from the engineering team that I found out the feature would be pushed back a bit into the 2.0 product. Bummer. Not to worry, they assured me it was coming.

6. When you click Return to Inventory list, you will see stats on the Netscaler. The most important thing to note is that green dot under State and the Status. That means it’s talking and getting AppFlow data. That’s it, you’re done setting it up! :)


1. Now if you went ahead and downloaded the 2.0 appliance, just skip this part entirely. But if you downloaded 1.0 like me and want to upgrade, this is how you do it. I’m assuming you have alread downloaded the NetScaler Insight Center 2.0 Build 112.13 upgrade pack I mentioned in the intro. It’s a 90 MB file called build-analytics-10.1-112.13.tgz. You’ll need to upload this to the appliance. In the Netscaler Insight web gui, navigate to Configuration > Netscaler Insight > Software Images > click Upload at the bottom and upload the image:


on the Documentation Files tab, you’re supposed to be able to upload the latest doc for the build just like a Netscaler appliance but I was not able to find the doc files anywhere on Citrix’s website. I am hoping they will add it to the Insight Center download page soon.

2. In the Netscaler Insight Center web gui, navigate to Configuration > System > System Administrator > Upgrade Netscaler InSight


It will automatically choose the image you uploaded. Wish I could find the documentation file that goes with it. I guess I can always update that later. Hit OK and it’s going to give you a confirmation prompt, just hit Yes to begin the upgrade.


3. It should be really quick. The confirmation that it was successful is easily missed because the upgrade dialog will still be open. It’s in the bottom left corner. When you click out your session should expire since the system is being rebooted:


4. Log back in and you should see you’ve been upgraded to Netscaler Insight Center 2.0 in the top left. The dashboard is going to look a little different, traffic is now categorized as Web Insight and HDX Insight. I can’t tell you how long I’ve been waiting to for that HDX Insight piece :) :


5. You can also go back to your device configuration and click on your NSIP and you should now see the VPN view in the drop down and any Access Gateway vservers you have. In this example I have just 1:


1. Right off the bat if you go back to HDX Insight on the dashboard, it’s not going to show any data. You might even get a popup error saying “this.chart_y_resources_property is null” or “this._info_cell is null” if you trying clicking around. First make sure you have enabled AppFlow on the Access Gateway vserver. You actually have to do this for any kind of vserver you have (LB, CS, or VPN) so you won’t get Web Insight data either until you enable it. You will get no metrics at all then. So go back to the device under configuration, highlight the vserver > click Action > Enable AppFlow. You’re going to get a screen like this to select an Expression. I chose SSL.REQ.VPN_VSERVER.NAME.EQ(“myvservername”) so I could grab everything. MAKE SURE to check the ICA checkbox or you will only see VPN traffic come across.


Once it’s enabled, you will see it say ENABLED under the Insight column. Here’s an example of an LB vserver I enabled it on:


2. Secondly, you need to be on NS 10.1 and have a Platinum license to really take advantage of Insight Center and get HDX metrics. NS 10 seems to work too but I have been doing all my testing on 10.1. Remember, Insight Center has requirements for both the firmware and license level of your Netscaler devices to function correctly. If you get error messages when clicking around, it’s likely due to this. I actually tried hooking up an NS 9.3 nc device but it did not work. It sort of worked with Insight 1.0 with web traffic. VPX Express devices are not supported with HDX Insight either since it’s a free license. I tried with an NS 10.0 VPX Express device during my testing and was unsuccessful pulling HDX Insight metrics. I wish Citrix would reconsider that one because I use VPX Express for testing config changes before applying on my real MPX and VPX devices and not seeing how my config changes are impacting test traffic is going to be a bit of a pain. I want the same visibility across the board from my lab environments to full scale production environments.

While I’m on the subject of feature requests, I’d also like to see the license level of each Netscaler displayed under the Inventory section. Currently it gives you a lot of great info on each Netscaler including the HA status and firmware version number but nothing about licensing. When you own a ton of Netscalers, it might drive you crazy trying to remember what license each has and what metrics are being reflected in the dashboard. If I were open up access to Insight Center to help desk or management, those users are not Netscaler engineers and don’t know the nuances of licensing. They might operate on the assumption that everything shown in the dashboard is all Netscaler traffic in the organization when it could certainly be missing a lot of data from non-Platinum devices and there is no way they would be aware of this. Again, the point of Insight Center is total visibility.

3. Once you’ve verified Insight is ENABLED and you are on the right firmware and license level, you may still not get any data. One thing to check is if your AppFlow port is open. Putty/SSH into one of your Netscalers and telnet to your Insight Center IP on port 4739 UDP (not TCP):

telnet 4739

If it doesn’t connect, there might be a firewall between the Netscaler and the Insight Center that may be blocking AppFlow traffic. Additionally it appears ports 80 and 443 are also being used. At the time of this writing, there is no eDoc available documenting the ports. I have requested that Citrix update this in the eDocs so hopefully we’ll get some official info online soon.

4. Verify the AppFlow expression and action were actually created on the Netscaler. I’ve seen some odd behavior already where Insight Center created a AppFlow policy but no action. Just log into your Netscaler and go to System > AppFlow > Policies and you should see all your policies there. You also have a hit counter so as you generate traffic, you should be able to see the hit counter rise:


You can also go to each LB, CS, or AGEE vserver as well as services/service groups and check there as well:



5. Make sure AppFlow is actually set to give the data Insight Center is requesting. On your Netscaler under System > AppFlow click on Change AppFlow Settings and make sure all the traffic you want to capture is checked. In my case for testing purposes I checked these. MAKE SURE ICA Traffic is checked:



6. I’ve had a few issues with Insight Center where I had to enable AppFlow on a vserver, disable it, then re-enable it to get traffic showing. Weird bug but you might want to try that as a troubleshooting step. I actually ran into the problem several times including right after doing a firmware update on a Netscaler. Disabling and re-enabling started pulling traffic again.

7. I ended up having to update Netscaler Insight Center to build Build 112.15 so it would work with a 10.1 Build 112.15 Netscaler appliance. Web Insight traffic works great. HDX Insight on the other hand is not working for me and I have had to open a ticket with Citrix on it. It is currently being escalated up to the engineering team. It seems Insight Center has trouble creating a VPN Appflow policy. Putty into your Netscaler, enter shell, and type:

cd /var/log


tail -f ns.log

then go the Insight Center and try and enable AppFlow on your VPN vserver. You’ll see an error in the ns.log when it tries to create the policy:

User nsroot - Remote_ip - Command "set appflow policy -rule "SSL.REQ.VPN_VSERVER.NAME.EQ(\"vsrv_xxxxxx\")" -action" - Status "ERROR: Expression syntax error"

Ok, so expression error. When you go to the Netscaler GUI and try and create the policy manually, SSL is missing. Probably the reason why it’s throwing a syntax error:


It’s being escalated with Citrix support right now. I’ll post an update when I get it. One thing to note, if you go to the firmware download page for 10.1 112.15 it now says in big bold letters “Access Gateway software in 10.1 release is a Tech Preview.” so that might be part of the problem.

8. One quick point on troubleshooting 10.1 112.15 Netscaler appliances. Trace logs can’t be opened in Wireshark 1.10.0 (the latest available). As you might know 10.x appliances require a certain version of Wireshark to view traces. Well with 10.1 there have been changes to the nstrace format and there is no “official” version of Wireshark that can view it yet. The changes were submitted by the Netscaler team to Wireshark and they will reflect in Wireshark 1.11.x when it is released. Until then the workaround is using the nightly builds of Wireshark which reflect the changes or using tcpdump.

Once you get it all configured and talking properly you’ll quickly see how invaluable Insight Center can be. Web Insight is working well and I can already see value in that. I’m hoping HDX Insight will be just as impressive when I get it working. I’ve heard we are just a few weeks away from a new Netscaler 10.1 firmware release so hopefully the kinks have been worked out. Insight Center might not be completely polished just yet but is it definitely headed in the right direction. I’m looking forward to future updates for this product. Please post a comment of your own experiences with it.

As of the June 27th Netscaler 10.1 firmware build 118.7, it has fixed the Insight Center issues I have reported. Here are the build 118.7 download links for the both the appliance firmware and Insight Center:

Here are the release notes:

And specifically, what I had reported:

“Issue ID 0388453: On the Configuration > Inventory > Application List page, after you right-click a VPN application and select Enable AppFlow, then clear the ICA check-box and click Enable AppFlow, AppFlow is shown enabled, but no data is collected and therefore no reports are displayed on the Dashboard > HDX Insight page.”

I also noticed this little bit which will make a lot of people happy:

“ENH ID 0395659: The ICA AppFlow records of NetScaler Insight Center were previously available only with Platinum licenses of NetScaler appliances. This release supports the Enterprise licenses as well.”

I take that back. Still experiencing some issues. HDX monitoring was working great for a bit but then completely stopped. After speaking with Citrix and again having to escalate to the dev team, I’ve been told devices with Enterprise licenses still do not pull HDX metrics and the release notes saying it does was an error. The August release will have the real fix. Known issues for the current 118.7 build are being logged here:

And here is the note about this issue:
Issue ID 0400900: The HDX Insight node is not displayed for Enterprise licenses of NetScaler appliances.

Getting real frustrated with Insight Center at this point. Especially since it was actually working for a while before it broke (I was taking screenshots of it working to put up when it broke!). I’m hoping the August release will be thoroughly QAed before being released. These half working 10.1 releases and contradictory information lately is making Citrix look bad in front of my peers. I kind of have a feeling 10.1 was rushed to market for Synergy and XenDesktop 7. I feel bad saying that because Netscaler is a stellar product with a good team of guys supporting it and I’ve never experienced issues like this in the past.

Latest version of Insight Center is super solid. Working exactly as expected!

How to setup Citrix Netscaler (Access Gateway) with multiple domains for web browsers and mobile devices

May 9th, 2013 8 comments

I’ve covered Access Gateway quite a bit in the past and these two articles on my blog are a good primer for what I’m about to cover:

1. How to setup your Citrix Netscaler (Access Gateway) and Web Interface for iPads and mobile devices that use Citrix Receiver

2. Quick way to brand Access Gateway and Web Interface 5.4 with company logo and colors

Now let’s throw a new scenario into the mix. You want to allow users from another domain access to resources through your Access Gateway and Web Interface. Here is what you need to do to accomplish this.

On your Access Gateway, if you followed my “branding guide” article above you will have noticed on the themes available from the Citrix blogs there are 2 versions, one normal and one with a dropdown. If you chose Horizon to customize like I did, make sure to download Horizon2.gz which has the code for the dropdown already in it. You just need to add your domains in the code. This is found in these two files:



If you have your own theme and just want to add the code, Citrix has an excellent CTX article here covering it:

How to Add a Drop-down Menu with Domain Names on the Logon Page for Access Gateway Enterprise Edition:

What’s happening here is that a dropdown box is created with pre-populated domain names that you specify in your code. The user selects the proper domain when logging in and then this domain name is inserted into the header and passed to the Netscaler Access Gateway vserver where the authentication policy examines it. Based on the name the Netscaler sees in the header, it forwards the request on to the correct authentication server. This works great with LDAP authentication servers. I have not tested it with TACACS or RADIUS yet.

Now it’s time to setup your Active Directory authentication for each of your domains.

1. Download LDAP Browser 4.5 for free here:

2. Go to your Netscaler under System > Authentication > Servers tab and create a new LDAP server. Use port 636 for secure LDAP instead of 389. Use LDAP Browser to get the Base DN and Administrator Bind DN. I prefer to use the app so I don’t fat finger the distinguished name and spend an hour trying to figure out that I missed a comma. Using LDAP Browser is much quicker and less error prone. The Base DN can be your entire AD domain or you can lock it down to an OU. The Administrator Bind DN is the actual account name that is used to enumerate AD. So CN = the user name, OU= the OU it’s in, etc. Add the password for this service account.

Make sure to select “SSL” for security type near the bottom. Click the blue “Retrieve attributes” link and it should talk to AD. Then the drop down boxes in Other Settings will not be greyed out anymore. Set the following:

Server Logon Name Attribute = samAccountName
Group Attribute = memberOf
Sub Attribute Name = CN
SSO Name Attribute = UserPrincipalName

So everything should look like this so far:


3. Now expand the Netsted Group Extraction arrow near the bottom. You need to enable it and set the same Group Name Identifier as you did for Server Logon Name Atrribute. In this case samAccountName. Then press OK to create the server:


4. Now it’s time to create the policy and bind it to the server. Hit the tab for Policies and create a new policy. Choose the server you just created. For the Expression, you will want:


where domain1 is the name of your first domain.


5. Now do steps 2 through 4 again but for your second domain, domain2.

6. Almost done. What we’ve done so far is create two authentication servers and two authentication policies. These two policies are good for web browser users (IE, Firefox, Chrome, Safari, etc.) but mobile devices using the Citrix Receiver native app like iPhones, iPads, Android phones, Android tablets, etc. will not work just yet. This is because they cannot pass cookie values. Nor do they pass the domain during authentication against the Access Gateway (a traffic capture can confirm it only passes user ID and password). You need to create a different set of authentication policies for these. So you will create one for each domain. You will bind to the same authentication server as the other policies but you’ll want to add the word “mobile” or something descriptive to the end of the policy name so you know what it is for. The expression should be:

REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver


7. Now do the same for your other domain, domain2.

8. Now go to your Access Gateway vserver and add all 4 policies you created. Take a look at my example screenshot closely and the priority order.


Here is the logic behind this policy order:

100 – Domain1 user logs in via web browser and uses drop down/passes cookie for domain1 so authenticates against domain 1 DC.

110 – Domain2 logins via web browser and uses drop down/passes cookie for domain2 so authenticates against domain 2 DC

120 – Domain1 user logins via iPad so first two policies are automatically false since the native Citrix Receiver app can’t pass a cookie. Authentication policy is looking for CitrixReceiver in the header so it will try to authenticate against domain 1 DC. It will be successful and the user will see his apps.

130 – Domain2 user logins via iPad. The first two are false for the same reason as 120. The expression from policy 120 is true so the user will actually authenticate and fail against domain 1 DC. Then it moves on to the 130 policy and authenticates against the domain 2 DC. It will be successful and the user will see his apps. Note, if a user has the same account name on both domains, it will cause a rejected login attempt on domain 1 which depending on your Active Directory account lockout policy can eventually lead to an account lockout. Just be mindful of this and adjust your policy accordingly if you need to.

9. Now on to the session policies. If you followed my guide on How to setup your Citrix Netscaler (Access Gateway) and Web Interface for iPads and mobile devices that use Citrix Receiver you should have 2 policies. One for regular traffic and one for mobile/Citrix Receiver traffic. Should look something like this:


Go ahead and click on the blue profile link for the CitrixReceiver policy first.

Under the Published Applications tab, clear the Single Sign-on Domain, you don’t need it anymore. SSO is being handled by the authentication server now. Specifically the SSO Name Attribute setting for each server which you set as UserPrincipalName. So it is automatically passing the UPN formatted user credentials to the Web Interface as the same time the user is successfully authenticating against the Access Gateway using the SAM account name. Pretty cool right? Now do the same for your other policy that controls regular web traffic.


11. An extra step for your mobile traffic policy. Under the Client Experience tab, you can set the Clientless Access to Allow and the Plug-in Type to Java but it’s not necessary. It will still work but since it’s mobile traffic only, this won’t hurt.


12. Now for your regular traffic, instead of using just 1 session policy as a catch all, you should create one per domain and set the SSO box so it passes the domain value. Otherwise you might land on the “agesso.aspx” page of your Web Interface with a “401 unauthorized: Access is denied due to invalid credentials” error message. Giving each of your domains a policy and forcing the SSO domain makes sure it gets passed every time to the back end web interface.


13. The correlating session policy for each domain that captures regular web traffic should check to see the User-Agent does not contain CitrixReceiver as well as look at the cookie value for the unique domain name. You will need to set the “Match All Expressions” option in your policies so it looks for both:


14. Now let’s move on to setting up your authentication groups. Let’s say each domain has a security group that gives users access via the Access Gateway. Again, there are 2 well documented methods for group extraction:

How to Configure a NetScaler Appliance for Active Directory Group Extraction for LDAP:

How to Configure a NetScaler Appliance for Active Directory Group Extraction for LDAP Using the Groups Allowed To Login Feature:

I prefer the first method. It’s basically just one step in one place (the authentication server object) and goes into effect at the authentication level. No fooling around trying to control it via a session policy. So go back to your 2 authentication servers you created before and take a look at the Search Filter field. It should be blank at the moment.

15. Go to LDAP Browser and get the DN for for the security group you want to give access to for your first domain.

16. Go back to the Search Filter field and type memberOf= followed by the DN (distinguished name) for the security group. So it should look something like this:


That easy. Make sure to do it for the authentication server for domain 2 as well. Go ahead and test your login scenarios and everything should work perfectly.

I only covered how to add a single authentication profile (domain controller) for each of the 4 policies on the vserver. Most companies are going to have multiple DCs. There is no way to add multiple DCs per authentication policy. The solution is to create a Load Balanced vserver with all your DCs behind it per domain and then add the IP of the vserver to an authentication server that is bound to the authentication policy. So in the example I’ve been showing you, I would have to create 2 load balanced vservers since I am working with 2 domains.

17. Go to Load Balancing > Servers > and add all your servers here:


18. You guys know I prefer to create service groups vs. services and the reasons why if you’ve read my previous Netscaler articles so go to the Service Groups section and add a new services group. Select SSL_TCP for the protocol.


Specify the server based members and make sure to set the port to 636. You can weight the servers here if you want to. Under the Monitors tab, go ahead and add a ping monitor or whichever monitor you prefer. Do this for each group of your authentication servers. You might want to group them by city, datacenters, production vs. DR, or whatever makes sense in your environment.

19. Now create a load balanced virtual server for your first domain. Select SSL_TCP for the protocol, port 636, and give it an IP address. Under the Service Groups tabs, add the groups you need. Keep in mind you won’t be able to see the SSL_TCP service group you created until you set the vserver protocol to SSL_TCP. Under the Methods and Persistence tab, you can get creative if you like or just leave it the default Least Connection. It really depends on your environment and where you prefer traffic to go. Under the SSL Settings tab, bind the cert you use for your Access Gateway vserver. Do all of this for your other domain as well. Both load balanced vservers should be in the UP state at this point.


20. Now go to System > Authentication > Server tab and left click on one of the DCs. Then at the bottom click Add, this will copy all of the settings on the DC you had highlighted and allow you to create a new one based off of it. So you don’t have to put all your DNs and stuff in again. Just give it a unique name and for the IP, type in the IP of LB vserver you just created for that domain:


Now do the same for the other domain.

21. Now go to the Policies tab and edit each of your 4 policies. You can simply change the server in the drop down to the new one you created for each of your 4 policies and you’re done. All authentication traffic is now going through your load balanced vservers. Keep in mind that before, LDAP traffic was going from your NSIP to each DC. When you use LB vservers, traffic is going from the SNIP to each DC. So make sure you have your firewall ports open from the correct source IPs or you won’t be able to authenticate. Retest all your scenarios and everything should continue to work just as before.

Once you’ve set everything up you might encounter issues and need to troubleshoot. There are a few ways to troubleshoot logins. I’ve covered this before here:

How to troubleshoot RADIUS or TACACS authentication issues on a Netscaler/Access Gateway

22. First you can troubleshoot the authentication layer but capturing of all authentication happening on the device. Open Putty and connect via SSH to your Netscaler. Enter the shell by typing


and pressing enter. Then type:

cat /tmp/aaad.debug

and the cursor will go to the next screen and wait. Open up your web browser and attempt to login to the Access Gateway. Immediately you will see the request happen in your SSH window. You’ll see exactly what is happening line by line like this. You’ll even see the nested group extraction taking place:


The last line will show the accept or reject:


When you’re done, press Ctrl+Z to exit.

23. The next layer to troubleshoot is all the polices. All of them, authentication, session, etc. You want to see everything a user hits when he logs in. So type the following:

nsconmsg -s disptime=1 -d current -g pol_hits

and login to the Access Gateway again using a web browser. Immediately in your Putty window you’ll see what all policies were hit:


This will help troubleshoot your policy flow. We’ve built a lot of intelligence into the cascading authentication and session policies and based on the user’s device and domain, it will vary what all they hit. This is an excellent way to test each of those scenarios.

24. The last step is getting down to the packet layer and analyzing the traffic using Wireshark. I’ve covered this in my previous post but go to System > Diagnostics > Start New Trace and set the packet size to 0. Then press Start.


Login to your Access Gateway via web browser, stop the capture, and download it to your desktop. You will also want to WinSCP into the Netscaler and grab the SSL RSA key for the SSL cert you are using on your Access Gateway site so you can decode the SSL traffic. Then double click on the .cap packet capture file and it will open in Wireshark. Go to Edit > Preferences > Protocols > SSL > and click Edit in the RSA Keys field:


In the SSL Decrypt Window, hit New in the bottom right side and this window will popup. Type in the IP address of the Access Gateway vserver, port 443, protocol http, the exact path on your hard drive to the SSL RSA key file, and leave the password blank. Then press OK to add it:


Press okay and get back to your capture. Now in the Filter field, just type the following:

frame contains youruserID

because we want to find all the instances of your user ID being passed. You should be able to find your user ID and password in plain text. Follow the TCP stream if you need to to find out exactly what is happening.

Hope this helps. Please leave a comment if this post helped you or if you have any questions and I’ll try my best to help. :)

How to create a XenApp 6.5 Server 2008 R2 golden image for PVS 6.1

April 24th, 2013 No comments

I urge you to read my PVS 6.1 on XenDesktop/Windows 7 guide first. Read it and make sure you understand it. The same concepts and most all procedures apply to Server 2008 R2. So I am not going to get as detailed on PVS technology in this article. This is going to be more XenApp centric. If you don’t have a thorough understanding of PVS, it is easy to become lost so please make sure and read that article and go through those screenshots first.

Citrix has an excellent eDoc primer for understanding the intricacies of XenApp on PVS located here. I recommend skimming this before beginning:

Now on to the steps. This is just my way of doing it. It works well for me. There are a couple of different ways you can achieve the same thing so use what works for you:

1. I am assuming you already have a XenApp 6.5 farm created and at least one dedicated server for the role of the ZDC. All XenApp servers provisioned via XenApp will be member servers of this farm and you do not want them to become a ZDC. Just set the election preference in AppCenter under Zones. In this example, I have a Default Zone and have set one server as the ZDC but it is best practice to have at least a handful of servers that are not provisioned to be set as preferred. Just in case one goes down, you don’t want one your provisioned servers to become a ZDC:


2. Now go to your XenServer and create a new Server 2008 R2 VM

3. Install XenServer Tools

4. Make a copy of the VM and convert it to a template. This is your “clean” Server 2008 R2 image you can come back to later if you need to. You can spin up new VMs from it.

5. Now go back to the VM you were working on and install things that are needed on all servers like Symantec, Citrix Offline Plugin (if you intend to leverage app streaming), etc. but try to keep it as clean as possible. Remember, you are building just the base right now. Don’t install any applications you plan to publish yet.

6. Add the server to the domain. Make sure the name is the first server in your naming scheme, example: “ServerName-100″ where 1 denotes the image number and 00 is the VM number. The next VM that gets spun up using this image will be ServerName-101 and so on.

7. Begin installation of XenApp 6.5 like normal and follow Approach 3 detailed here:

8. If you have multiple NICs in your PVS environment for streaming vs. regular network traffic there’s an extra step you have to perform in the VM. Make sure you go into ICA Listener properties and set it to PVS Adapter #1, the network NIC. Not the PVS streaming NIC which is #0. They may be labeled differently in your environment depending on the NIC order of your VM. Just remember, you want the network NIC to handle ICA, not the streaming NIC.

9. Now install the apps you intend to publish. Just install, don’t try publishing anything yet. If you plan on streaming apps to your XenApp servers via Citrix Offline Plugin or App-V, skip this step.

10. Install the PVS Target and reboot. Do not run Imaging Wizard yet.

11. You can choose to install the EdgeSight agent at this point if you use EdgeSight in your environment. There is an excellent guide from Citrix on how to install EdgeSight in a PVS environment below. Don’t worry, when it detects the PVS Target software is on the system and the image is in private mode being updated, it won’t start the EdgeSight service and start sending your EdgeSight server junk data:

12. Log back in to the VM and in the XenCenter console, set the PVS boot disk to boot first (BDM). Then go to the PVS Console on the PVS server and create a new Device with the MAC address of this VM’s provisioning/streaming NIC. Call the device ServerName-101 so it’s separate from the ServerName-100 original but call the vDisk 100 so you know where the image came from later (i.e. image 1). Set device to Boot from Hard Disk. Now reboot the VM. You can let Imaging Wizard do this but if you really want to be hands on, you can choose to create and attach an empty vDisk of the same size as the c: drive of the VM now.

13. It should have booted from the BDM boot disk and be connected to the empty vDisk you created. Run Imaging Wizard and push the image across to the PVS Server vDisk (don’t forget to optimize the image). If you did not create a vDisk in the above steps, just create a new disk and run through the Wizard prompts to create it real quick. It will assign the device to this new vDisk. Reboot when you are asked to.

14. Log back in and immediately XenConvert will launch and begin pushing your image over to the newly created vDisk.

15. Now go to the PVS console and set your device to boot from vDisk and reboot your VM. It should now be booting from your vDisk.

16. Log back in and re-run XenApp configuration and unjoin from farm using the prep option. This is done by going to Start > Programs > Administrative Tools > Citrix > XenApp Server Role Manager > and then clicking on the XenApp Server Role Manager app. Click Edit Configuration. Then click Prepare this server for imaging and provisioning. You want to just leave the defaults checked which is to remove it from the farm and join on the next boot. Once it removes it, it will ask you to reboot. Just close and power down the VM, don’t reboot.




17. Go to your PVS server and change the vdisk from private to standard mode. Now spin up a few VMs using the “Streamed VM Setup Wizard” in the PVS console. You’re going to need a template and you’re going to want to setup your write cache. I’ve covered this very well in my PVS 6.1 on XenDesktop/Windows 7 guide so I’m not going to get into the details here. Wait until the VMs are created and powered on.

18. Go to your ZDC and open up AppCenter and run a Discovery. You should see the new member servers. At this point you can begin publishing or streaming apps to them.

1. Boot a device into maintenance mode after creating a maintenance version on the vDisk. Or you can choose to power down all your running VMs and put the disk into private mode. Your choice.

2. Make your changes.

3. Re-run XenApp configuration and unjoin from farm using the prep option.

4. Shut it down and promote the image to production if using versioning. Or put the disk back into standard mode if you used the other method.

5. Reboot all your devices so it gets the new vdisk

I hope this guide helps. Please feel free to leave a comment below if you have any questions. I’ll try and help as best as I can. :)

Opening Sharepoint documents in a Windows 7 XenDesktop session using XenApp published Office products with content redirection

March 26th, 2013 2 comments

Wow, long title there! Just like it says, I’m going to cover opening Sharepoint documents in a Windows 7 XenDesktop VDI session using XenApp published Office products with content redirection enabled. Here’s an example scenario:

  • Your company uses Sharepoint and the URL is http:\\ It is part of the Local Intranet zone in IE.
  • Your users are using Windows 7 VDI/XenDesktop VMs with Citrix Receiver Enterprise pulling published apps into the Start menu.
  • You have Office 2010 products (Word 2010, Excel 2010, Visio 2010, etc) published via XenApp with content redirection enabled so .docx, .xlsx, .vsd files launched in the VDI session will open using the published Office app.
  • Your users want to click on a Word, Excel, Visio, etc. document in a Sharepoint document library and click the Edit button to edit the document instead of just Read Only. They expect the published Office app to launch and allow them to edit the document.

In this scenario, when the user clicks on let’s say a Visio document in Sharepoint, the published app will launch but the document will not be sucked in. You’ll get a “File not found” error and if you look closely, the URL begins with “\\” as if it’s a UNC path instead of an HTTP path:


If you click OK, Visio will open but no diagram will open.

If you right click, Save As the Visio diagram to your C: drive and attempt to open it locally, it will launch the published Visio instance but will give you the following “File not found” error:


Again, look closely. It says “\\Client\” instead of your VM’s name.

Well, really it’s 1 fix and 1 workaround for now. Let’s start with the fix first. We’re going to get local files and files residing on network shares launching right using the published app. I’m going to use Visio and a .vsd file again as the example:

1. On the Windows 7 VM, you must enabled Native Drive Mapping to be able to launch the .vsd file via published Visio. You can follow the CTX article here to make the registry change:

or you can enforce it via group policy. One thing though, on Windows 7 x64 the path is:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\ICA Client\Engine\Configuration\

Reg Type: REG_SZ
Add the Value: True

Once the value is set to TRUE, try launching the .vsd file you saved to your hard drive or on a network share. It will launch immediately. Then go to File – Save As in Visio and checkout the path bar. It will say “\\Client\C$\Users\xxxx\”. So now your system recognizes the word “Client”.


This is because you’re going through 2 virtualization layers. Citrix has an excellent article detailing how Client drives are passed through multiple sessions here:

You can also read more about the NativeDriveMapping key here:

Additionally here is a really good CTX article from Citrix that covers Client Drive Mapping troubleshooting that might be helpful to you:

And some info describing differences in Client drive mapping between the different versions of XenApp and Presentation Server here:

2. Now back to Visio diagrams in Sharepoint. First you need to verify content redirection is behaving correctly. In my case, I needed to make sure to change the published app location to include the “%**” switch. By default when you enable content direction, it will be:

"D:\Program Files (x86)\Microsoft Office\Office14\VISIO.EXE" "%*"

with just one asterisk. You need to add two so it looks like this:

"D:\Program Files (x86)\Microsoft Office\Office14\VISIO.EXE" "%**"

The reason for this is documented in this CTX article:

This was written for Presentation Server 4.5 but it still applies to XenApp 6.5 just fine.

Once you get content redirection squared away on your XenApp server, things will still not work correctly unfortunately when opening docs in Sharepoint in your XenDesktop session. You can try right clicking and editing but you will still get the File not found error. This error has nothing to do with Client Drive Mapping. Being a Sharepoint Architect in a previous role long ago, my gut feeling was this is a WebDAV issue. Remember, Sharepoint uses WebDAV with document libraries though to the casual observer it might appear to be a UNC file share path. My gut was telling me Receiver was not handling WebDav paths correctly and was treating them like regular SMB file share paths. No proof this is the culprit just yet, only a hunch.

Let’s run an experiment. In your Sharepoint document library, click Actions > Open with Windows Explorer:


You’ll notice the path bar says “” when you open it here:


Go ahead and launch the Visio diagram in this document library. When you try and launch it, it will launch published Visio and give you the same File not Found error as before. In some scenarios even an Access Denied error though it never actually was able to find the file. Again, it is looking for “\\” instead of using WebDAV (or the DavWWWRoot keyword) which should be parsed like the HTTP protocol “http:\\”. Sharepoint Content Redirection is never going to work because Citrix Receiver doesn’t know how to get there. One work around is to tell your users to save locally, edit, then upload. But this is annoying and most users will complain.

I started thinking Mini Redirector (which is Microsoft’s WebDav client and part of the Windows 7 OS) was trying to hand-off the DavWWWRoot keyword in the URL to Receiver and Receiver didn’t know what to do with it. Looking at the URL though it didn’t even seem to be passing it through. It was as if it was passing a straight UNC path. Or maybe is was not talking to owssupp.dll which is required to interact with and edit docs in Sharepoint. Maybe somehow it wasn’t handing off through Receiver properly. Now I was really starting to go down the rabbit hole and before I dug further, I figured I’d give Citrix Support a call and see if they had any input.

I was able to talk to a XenApp support technician and a XenApp Developer who were both very knowledgeable. I explained my WebDav theory and we ran a little experiment. We ran Process Monitor from my Windows 7 workstation and attempted to edit a simple Word document in Sharepoint. The published version of Word launched as expected and failed to find the file. Here is the URL that was passed to the PNAgent when we used the Process Tree display:


You can see that it passed:

\\Client\\\\link\to\doc\repository\test jason.docx"

Looks a bit odd, doesn’t it? No wonder it can’t find the file.

Then we went to the XenApp server with Word 2010 installed locally, opened Internet Explorer, and navigated to the Sharepoint document library. When I clicked Edit on the Word document this time, this is what was passed to the locally installed Word:


And of course the Word document successfully launched. You can see it passed a nicely formatted HTTP URL:"

The XenApp Developer verified this was actually a bug and needed to be fixed. It may or may not be WebDav related, but he confirmed there is definitely a disconnect at the Content Redirection level via Sharepoint as I had suspected. It has been added to their bug queue to get taken care of.

In the mean time, his suggestion is to publish Sharepoint itself on the same server Office 2010 products are installed on (publish IE passing the Sharepoint URL during launch). This is a good workaround but may annoy a few users. I always hate publishing IE because of the stuff you have to do to lock down/harden the browser. People can get confused and start browsing to Youtube or other process intensive websites using this published instance of IE if you don’t completely lock down the allowed URLs. This can cause a severe impact on XenApp performance for all users on the box. If I limit tabs, then people start complaining they can’t open additional tabs and the helpdesk gets flooded with phone calls. So just keep that in mind if you use this workaround.

Another work around I found online is by Anthony Obi:

He has an issue where owssupp.dll was not being detected on the client system (because it wasn’t there) and added an extra menu item in his Sharepoint deployment to edit documents as a work around. He modified the new Edit action to launch some Javascript that talks to PNAgent.exe that then talks to the published Office instance and passes the correct URL. This might work as well for you but of course it requires a modification of your Sharepoint environment and end user training.

I will post here as I get updates from Citrix Support on this issue. It’s a fairly unique scenario but definitely not uncommon in the real world as application delivery and desktop virtualization continue to be separated through multiple layers.