Are your users blown away by how fast VDI logins or are they horrified at how slow it is? If horrified then read on. Folder Redirection and GPOs are necessary evils of running a pooled VDI environment. A lot of people like to concentrate on storage, master image, or PVS and hypervisor tweaks to resolve performance bottlenecks. Those are all important parts of the VDI stack and it’s true, they can be the cause of bottlenecks. But the reality is bloated GPO and folder redirection are the bigger culprits in most environments I’ve seen and people often don’t think about them because they are not as visible as the other bottlenecks.
You can easily resolve the other bottleneck issues like storage by throwing money at the problem most often than not (SSD, flash, etc.) but GPO and folder redirection are things you have to spend a lot of time on. You have to have intimate knowledge of the environment and really know what you’re doing. Above all you have to break out of the physical world mentality. Time and time again I’ve come across environments that are still applying physical desktop mentality to virtual desktops and it never works out well for them. Lastly, once changes are made they never bother to poll their users and see if there really is a perceptible change in the environment. I’m going to cover a few things in this article:
- Why is Folder Redirection & GPO so evil?
- How AppSense can help with GPO
- How AppSense can help with Folder Redirection
- Catching Folder Redirection issues & preventing data loss
- Getting user feedback on their VDI experience
WHY IS FOLDER REDIRECTION & GPO SO EVIL?
Folder Redirection and bloated GPOs are not as visible so it makes it a little harder to troubleshoot. Once you really start analyzing your environment you might find that the hardware and image are really performing very well, it’s the external stuff unrelated to VDI that is slowing you down.
Folder Redirection and home drives are necessary for pooled VDI because how else are you going to get the user’s data into the system and back out since it will not persist a reboot? A burned in mandatory profile in the master image + folder redirection seems to be the defacto standard for pooled VDI from my experience. There are other solutions to bring the user data in like 3rd party file synching (Citrix ShareFile with On-Demand Sync, etc) but none of these integrate to the Windows Shell, Office suite apps, IE, etc. as well as the native experience does. So you’re pretty much forced into a corner and have to use Windows Folder Redirection for the best physical desktop like experience. Problem is if your really expensive VDI storage solution is stellar and getting 60,000 IOPs per VM you still have to rely on this external file server somewhere with spinning disks to deliver a portion of the OS to the user across the LAN and any slowness there makes the whole VDI experience look bad. Oh wait, did your Antivirus guy set some kind of exception on this traffic or is it getting scanned? Is it getting double scanned, once at the file server and again as it comes into the Windows 7 OS? Oh wait, your network guy just implemented an appliance level malware scanner and it’s scanning this traffic too now? Great.
File shares weren’t really meant to deliver a portion of the OS in my opinion. If you’ve ever used AppSense Environment Manager and hiving, you know all those little registry keys being stored out there have to be on some pretty fast storage with the least amount of resource contention on the file server for the best experience. The SMB protocol was not designed for this in my opinion. Sharing files vs. delivering parts of an OS are two entirely different things with different performance needs. If you take a look at the contents of any hiving you might being doing you will probably find it’s composed of a bunch of tiny little files that need to be loaded very quickly into the system with very critical timing considerations. If any of that is slow, user experience is impacted negatively.
Let’s not the forget about the other evil thing hidden in everyone’s environment, GPOs (Group Policy Objects). Over time GPO becomes bloated. The guy that stood up the very first desktop policy is your environment 15 years ago when your Windows Active Directory domain was created is no longer around and since then 12 other guys have come and gone and had their hands on it. No one person on Earth knows what any of it does anymore. There’s a bunch of logon scripts being called on, are all those doing anything useful? There’s a bunch of super slow WMI filtering going on but nobody really knows why. And then there’s that one off in there to fix that one guy’s laptop issue from 5 years ago. But it’s been compounded by even more one offs over time. No no one knows if it’s safe to remove any of it. Is that guy and his laptop still even around?
Each and every one of these items in GPO has a penalty. The penalty is in the form of time and compute. The more items in your policies you have, the harder your system has to work and the slower your logins will be. Check your CPU and RAM usage at a user logon and you’ll see them spike close to 100% for a little while in a really bad environment. Seems like it takes minutes for UserInit.exe to end (FYI, this is the Windows process that runs logon scripts, starts explorer.exe, etc and until this ends, your desktop is not truly ready to be used). The solution in the physical desktop world is to mask these issues. Throw a quad core CPU with 8 GB RAM and an SSD under everyone’s desk and call it a day. Brute force through the logon issues so users don’t complain. But this approach doesn’t work too well for VDI. Virtual Desktops are given a sliver of shared compute on the hypervisors so any issue that is masked like this on physical desktops by brute forcing becomes highly visible in VDI.
HOW APPSENSE CAN HELP WITH GPO
AppSense Environment Manager can help you reign in your GPOs. But it requires a lot of work on your part. You need to first log into a virtual desktop session, open a cmd prompt, and run a:
gpresult /h gpo.html
and view the resulting gpo.html file to see what is hitting your VDI infrastructure. Keep in mind that though machine policies will be the same for you, user policies can be different from one user to the next. That’s why I always recommend setting Group Policy loopback processing mode to “Replace” on the OU your Virtual Desktops are in. What this will do is is replace all the nasty group policies being applied elsewhere at ther user account level with only the ones you are about to specify at the OU the virtual desktops are in.
Now once you have a general idea of what’s being applied, your goal is to now move all the user related policies into your AppSense Environment Manager config. This is pretty straightforward. Group policy is nothing but a series of registry edits done to the PC for the most part. AppSense can read those same GPO templates you use and apply the same registry edits. Only faster. You see, group policy is single threaded. It all runs one right after another. AppSense EM on the other hand is multi-threaded and runs through all the polcies at a fraction of the speed native GPO does. I’ve literally seen logins that take over 1 minute come down to the 10 second mark. It works very well but you have to invest the time to fine tune it. Trust me, it’s worth taking the time to do it right.
AppSense EM is one of those things where you really need to know how GPO works to effectively use it or you will be lost. That’s why you have to spend time to really learn the results of your gpo.html file. Print it out, pull out a highlighter, and highlight only the things you need. If you don’t know what something is, leave it out. The less stuff you have in your environment the better it will run. Trust me on this. If you want to get crazy and lock down stuff do it later. Just to get the ship into the water I recommend running the leanest EM config you can with a supporting single GPO (aka Baseline GPO) applied at the OU your virtual desktops are in. In this single GPO I would put a few machine level things (Computer Configuration policies) and that’s it. The rest is all handled by AppSense.
HOW APPSENSE CAN HELP WITH FOLDER REDIRECTION
AppSense Environment Manager (EM) can help with folder redirection. My recommendation is to make sure all your users have security groups that define where on the network their home drive file shares are located. If that doesn’t happen it’s quite possible if you try to programmatically map it at logon with AppSense or other method, some people aren’t going to get folder redirection and that’s VERY VERY bad. More on this later.
Your AppSense EM policy config has a node called Pre-Session under the Logon node:
This is basically stuff that is set first during the Logon process. Here you would set your “AppSenseHome” environmental variable to something like:
where all hiving will go. This is common for all users and needs to be as close to the location of the Virtual Desktop as possible. Remember, a good portion of the operating system can come from here depending on your setup so don’t skimp here.
Next you want to setup your “HomeFolderRoot” and build the necessary logic around your security groups I mentioned above. So:
etc. for all the different security groups. So in this example the security group “SG_HomeDrives1” will get mapped to to the “homedrives1” location, “SG_HomeDrives2” will get mapped to the “homedrives2” location, etc.
Then further down in your Pre-Desktop node, you can begin to setup your policies for the Folder Redirection. In most environments I’ve seen people like to redirect these at the least:
%HomeFolderRoot%\My Documents\My Music
%HomeFolderRoot%\My Documents\My Pictures
%HomeFolderRoot%\My Documents\My Videos
so basically what you’ve done is centrally mapped the security group to the proper home drive location during Pre-Session. Then later AppSense during the Pre-Desktop it maps the redirected folders above seamlessly based on the location where their home drive resides. Pretty cool huh?
I also recommend putting in the Duplicate Folder fix for Redirected Folders in this node.
Hexadecimal Value: 00000001
as well as turning off the message about Windows Library Features when using Redirected Folders. That gets annoying real quick but thankfully MS has a GPO template for it and the setting is “Turn off Windows Libraries features that rely on indexed file data”:
CATCHING FOLDER REDIRECTION ISSUES
Ok, so what happens if the user isn’t in a security group for whatever reason? Then the user would login, all the shell folders above would point to the local C: drive, the user would save a bunch of stuff to My Documents, and when they logout and log back in all their stuff is gone. Great.
Seriously, this is your worst nightmare. Luckily AppSense EM policy is pretty extensible. I always like to write a few checks into my AppSense EM config to cover this and a few other scenarios which I’ll share with you. All this happens automatically at the user logon for every session:
- Check to see if the user is a member of a home drive security group
- Checks to see if the user can create a “My Documents” folder (this tests Read/Write permissions on the file share)
- Notify the user if any of these checks fail and let them know IT is aware of the problem
- Log user off immediately to prevent them from using a broken virtual desktop and possibly lose data
- Notify the Help Desk for resolution
Then let’s make sure a My Documents folder and AutoRecover folders are being created. What this does is verify the HomeFolderRoot location can be read and written to by the user. Again add a bit of logic to detect if it exists or not:
Now let’s move on to your Desktop Created node. Create a new node called “Verify Folder Redirection”. Now build your logic out. I’ll explain my logic line by line. First I want to exclude certain accounts from being checked and in the registry, I want to create a key saying that folder redirection was not checked for that user. You always want to leave behind these little keys that you hive out in case you need to troubleshoot something:
It’s just a bunch of ORs for admin accounts, monitoring accounts, etc. that don’t utilize folder redirection like this. You can explicitly define the accounts or do a match (wildcard) for the user name:
Now on to the Else. First we need to check if the HomeFolderRoot does not exist. If it does not exist, that likely means the user was not put into one of the security groups you defined earlier. If this is the case, you want to popup with a message letting the user know that something is wrong and IT staff is being notified they are having an issue:
The parameter you want to pass is something like this:
* /v Your folders are not being redirected. The Help Desk has been notified. You will be contacted by IT personnel within 24 hours. Your Virtual Desktop will now shutdown within 10 seconds.
The intent is that once you deploy the config and the user logs in normally, if everything is good they’ll be able to use their virtual desktop and be none the wiser. But if any one of the issues above is detected they will get this prompt. They won’t have the opportunity to use their virtual desktop at all and potentially lose their data due to broken folder redirection. They’ll be be logged out within 10 seconds before they have a chance to do anything like save an important document locally. Pretty cool right? Getting kicked out is a minor inconvenience for the user but will save both them and you a lot of heartache over a lost spreadsheet as well as notify IT staff there is an issue. All automatically. 🙂
As part of the Else, you want to actually do what you tell the user you are going to do. You want to email staff and raise some alarms here before shutting down the virtual desktop. You can email your help desk, VDI/AppSense admins, security admins, AD admins, IT managers, ticketing system, reporting log, whatever. It’s just a Powershell action:
Here’s my PowerShell script that does the heavy lifting. Feel free to modify as needed:
$UserName = (whoami)
$smtpServer = “yoursmtpserver”
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$msg.From = “firstname.lastname@example.org”
$msg.Subject = “Folder Redirection broken for ” + $UserName
$msg.Body = $UserName + “ has broken Folder Redirection in their Virtual Desktop. The user was notified what is happening and their Virtual Desktop was shutdown after 10 seconds. This email is BCCed to the Ticketing System for ticket creation. Please assign the ticket to the Security team. The user will continue to be blocked from using their Virtual Desktop until the issue is resolved. This is an automated email from AppSense."
and the node below that is yet another Powershell action:
it’s just a a simple restart command to reboot the computer after 10 seconds of displaying the message to the user:
Start-Sleep -s 10
The last bit you need to add is if the My Documents folder exists. This means all checks have passed and the user can use the virtual desktop normally. All I do is add a little registry key letting me know their folder redirection is working as a record if I ever need to test something with them in the future:
GETTING USER FEEDBACK
Once you’ve built this wonderful VDI experience with great login times you need to get feedback from the users. If you build $1 million dollar VDI infrastructure and no ones uses it because of one tiny issue you glanced over, your whole project can fail. User feedback is critical for any VDI implementation. If you spend the time to make changes in your environment and improve the user experience, you must poll your users and see if they notice this change and whether it’s a positive or negative change. Any feedback you can get from people is a good thing.
My suggestion is to quickly create a simple website with a little bit of info on how to use VDI, some how-to videos, a PDF guide, etc. This ensures expectations are set and the user is trained on how to use the environment. You also want to create a separate URL for a feedback form that you can present to the user once the user has had a chance to use their virtual desktop for a while. If you’re not adept at creating websites and content, you can pop-up a Word or PDF doc instead. For user feedback, you can simply point the user at something like SurveyMonkey once you’ve created a feedback survey form on there. There’s so many ways you can do this so don’t limit yourself and say it’s too hard to do. In this example, this is what I want to happen:
- 1st login = User gets Welcome Message & instructions on how to use VDI properly
- 2nd login = Request Feedback from the user on their VDI experience
In this example I’m going say you have 4 master images you are running off Citrix PVS and there are 1000 machines spun up for each (so grand total of 4000 VMs up). For the first 2 master images, all the VMs start with “VM1” or “VM2” denoting image 1 and 2. The next 2 images starts with “VM3” and “VM4”. I want to leave the first 2 images alone and only display a Welcome Message letting users know how to use VDI properly and get Feedback for the second set of VMs (so 2000 out of the 4000 VMs).
Yeah, I know. The logic looks pretty funky but it works. Let me walk you through it. First I want to make sure the message is only displayed on my VM3 and VM4 VMs:
Here is my IE script. I like to prevent the user from actually using the browser for anything but what it was popped up for so I block out all the tool bar (remember my Publishing IE via Citrix in full screen kiosk like mode and retaining the X button to close article). Think of it as a bootleg kiosk mode. Again, it’s just a simple VBScript:
Set objIE = CreateObject("InternetExplorer.Application")
iHeight = .height
iWidth = .width
objIE.StatusBar = False
objIE.Visible = True
objIE.AddressBar = False
objIE.MenuBar = False
objIE.ToolBar = False
objIE.Top = 0
objIE.Left = 0 + 8 'Move to the side just a bit to show the desktop
objIE.Width = iWidth - 16 'Shrink to let a bit of the desktop show on the sides
objIE.Height = iHeight - 28 'Shrink a bit to see the taskbar
The last bit of the logic adds the registry key value “WelcomeMsg3and4” in the user’s RunOnceChecks that you hive in and out. This is basically a counter. It shows that the user has gotten the Welcome Message exactly once. You don’t want to keep showing the message over and over again each time the user gets a new VDI session:
Moving on to the ELSE statement. Now we start looking for that registry key on these 2 sets of VMs. If it exists, then let’s give them the Feedback Message. It’s just the same IE script above but with a different URL that shows the Feedback Message:
Again, it doesn’t have to be anything overly complicated. Just a simple text box like in my example screenshot below is sufficient:
If you don’t want to create your web content from scratch, pointing at a SurveyMonkey link is fine. You can do open ended questions with a text box like above but I find people respond more when presented with per-populated radio box options like SurveyMonkey does. Users don’t want to think. They want to be given a few choices, click on them quickly, then move on with their lives. So try and make it easy on them. Max 2-3 questions. No one wants to fill out a 20 question survey unless you offer them something in return like being entered in a drawing for an iPad or an extra vacation day. Incentivizing surveys is a very powerful technique to boost user engagement (just be aware it can skew results too, the guy that gives you all 10s so he can get a day off with his new iPad).
We also want to leave a registry key as a counter letting the system know the Feedback Message was display once to the user. You don’t want it to popup over and over again. The next time the user logs in after getting the message the logic is run through very quickly but nothing will popup again. You can continue to add little IE window pop ups for future messages to your user base this way. Pretty cool right? Big thanks to my friend Paul for helping me work out this logic. 🙂
For those of you that have been working in the VDI world for some time, you know the pooled VDI stack is pretty deep and complicated. Just when you think you fixed one bottleneck, another one appears. Just when you think you’ve covered all the layers, one starts to unravel and bite you. Most often than not from my experience it’s been Folder Redirection and GPO issues that have been the big cause for poor login performance in VDI environments. Hopefully some of the ways I’ve showed you how to mitigate these issues using AppSense Environment Manager will help you out.
Please leave a comment if you have any questions or have other ways of mitigating these issues that you have implemented. Always good to hear more real world experiences. 🙂