1/19/2012 x:xx:xx PM Error: Starting VM 'xxxxxx' - Storage assignment failed
(SSL_ERROR_SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
verify failed)

in the XenCenter log. You can verify the SSL cert by opening up the following in a browser window and replacing the x’s below with your StorageLink server’s IP address:

https://xxx.xxx.xxx.xxx:21605

You will get a cert error message in your browser. Notice the Citrix CVSM SSL certificate issued on 1/19/2009 has expired today 1/19/2012 at 20:25:53 PM (GMT) which is 2:25 PM Central Standard Time. So basically any VM you try to turn on, reboot, or migrate after the cert expired will not work and return the SSL error above in the XenCenter log. Yeah, big problem.

I was the first to call in about this issue soon after the cert expired apparently. As I was on the phone troubleshooting this with the support engineer, others began calling in with the same problem. We have escalated it to the highest level at Citrix support and have been assured a workaround and a new cert are both being worked on and something should be available tomorrow morning. This is going to impact pretty much all StorageLink customers globally so trust me, they are working on it. Over the past several hours, I have tried numerous workarounds myself but been unable to get a full fix yet. I’ve tried self signed certs using OpenSSL, IIS & SelfSSL, etc. but to no avail so far. The StorageLink Gateway does not use a web server such as lighttpd, Apache, Tomcat, etc. either so I can’t force it to use another set of certs on that end. Apparently it uses API calls. When you restart the services, you will notice it copies the following SSL certs which are the culprits (into memory I’m guessing). I used Process Monitor to verify:

D:\Program Files (x86)\Citrix\StorageLink\Server\cacert.pem

D:\Program Files (x86)\Citrix\StorageLink\Server\server.pem

I actually did manage to get a little further than I thought on the handful of workarounds I tried, but nothing completely successful yet to regain functionality while we wait for a hotfix. If you want to try playing with the certs yourself, just remember to restart the StorageLink services after you swap out the certs each time so it pulls them in. XenCenter should see the SSL cert change and prompt you almost immediately with a warning message.

I will keep this post updated with the latest developments. Please post if you are having the same issue or come up with a temporary fix. In the meantime, call Citrix and open a case so you are in the loop when the fix is released.

UPDATE January, 24th, 2012 – Citrix has published the fix:

http://support.citrix.com/article/CTX131994

Apply the certs using the instructions in the KB. Shouldn’t take long at all.

Citrix XenServer and StorageLink SSL cert error caused by expired SSL certificate is a post from: JasonSamuel.com

More of my posts you might like:

1. How to create and link an SSL certificate chain for a Citrix Netscaler
2. How to properly use SSL redirects without getting certificate error messages
3. SSL certificate chains explained

This is all a pretty good sign the XAPI service on the host is hung up. XAPI is the XenServer management toolstack which pretty much controls everything on the XenServer host. If the “XenAPI” toolstack is hosed, XenCenter can’t talk to the host and you probably won’t be able to pass any xe commands. The Xen API is what controls everything at the host layer. Quick way to troubleshoot this:

1. SSH into the host with the issue.

2. Type:

df -h

which will show the disk space usage on the file system. The “-h” switch will display it in gigabytes. Much easier to read. We need to check the root partition and see if it is full. This is typically 4 GB and can be filled up by logs which may cause the XAPI service to stop. If the XenServer root disk is full, you will probably see it drop out of XenCenter because XAPI is stopped. You won’t be able to restart the XAPI service until you free up some space. Here is an example of the root being 100% full:

Extra tip, once you log in to one XenServer host, you can check other hosts remotely without having to SSH into each one in a different terminal. Just type:

ssh <RemoteXenServerIPorName> df -h

3. If the root is full like above, type:

cd /var/log

then

ls

to list the logs. Type:

du –ksh *.*

to list the logs with the sizes. If you find one that is too big, delete it:

rm <logname>.log

From here you can skip ahead below to step 6 and try restarting XAPI.

Also, you might want to consider moving your logs off to a different volume. If you fill your dom0 root, you’re basically hosing the XenServer. Citrix has a good article on how to move the /var/log directory to a different volume here:

http://support.citrix.com/article/CTX130245

or retain fewer logs by editing logrotate.conf here:

http://support.citrix.com/article/CTX131619

4. If your root is not full, the next thing you probably want to do is disable HA. You can do this in the XenCenter console or you can just type:

xe pool-ha-disable

or if you want to disable HA on a host (you’ll have to run this on each host though):

host-emergency-ha-disable force=true

5. After disabling HA, restart the toolstack:

xe-toolstack-restart

This will disconnect all the hosts in the pool in XenCenter but don’t panic. Give it 10-20 seconds, once the toolstack is restarted the hosts will all reconnect to XenCenter. All pending actions like reboots, migrations, etc. will all stop when restarting the tool stack so you have a clean slate.

6. You should be able to console into your host with the issues now. Type:

service xapi status

and see if it is running. If you want to see how taxed XAPI is, type:

top

to see all the running processing. If XAPI is taking up 40% CPU or more, that is a good indication something is hung up on it.

If XAPI is not running or is very taxed, type:

service xapi restart

if it hangs at “Stopping xapi” or “Starting xapi”, you may need to kill the process.

Type:

kill <pid>

using the process ID from when you ran “service xapi status” or “top”. Then service xapi status to verify all xapi processes have stopped. Then you can type:

service xapi restart

again if it didn’t automatically try and start already. Eventually it will say:

Starting xapi: ....start-of-day complete.                  [  OK  ]

and you should see the host pop back in your XenCenter console. If you go back and run top, xapi should be taking up around 1% or less CPU.

You can type:

xe task-list

to see all the running tasks which shouldn’t be much at this point. Don’t forget to re-enable HA after you’re done. Hope this helps someone.

Can’t console to frozen XenServer host but VMs are still running is a post from: JasonSamuel.com

More of my posts you might like:

1. Running Open Solaris on Citrix XenServer 5.5 shows no IP address associated with the VM
2. How to use an external USB drive as a storage repository in XenServer 5.6

A lot of what was covered is stuff most everyone working with VDI may already know. I’m not going to transcribe all my notes from the seminar. But some key takeaways that I felt were important were the following:

1. VDI projects and hardware refreshes are driven by different factors than last year this time. April 8, 2014 (End of Support, patches, hotfixes, etc. for Windows XP) is looming and many IT departments are concentrating more on hardware/Windows 7 migrations than they are on VDI projects. Last year many companies were trying to do both since these projects usually go hand in hand. Many companies that are still doing POCs or extending their testing phases are now starting to feel a time crunch. Because of this, many companies are giving priority to the refreshes and letting the VDI projects become a secondary project. That’s not to say VDI is not important, it’s just that many companies need to prioritize resources and personnel and choosing to allocate these at the refreshes. It’s funny because this has been happening all around me and to friends at their companies but I didn’t fully realize how industry wide this shift of priorities was until Brian said it out loud. So the takeaway is, get your refreshes completed first if you don’t have enough resources to handle both. You DO NOT want to get caught in 2014 with any XP machines on your network or something seriously needs to be reevaluated at your company. And it may end up being your job if you don’t drive the initiative to get your refreshes done well ahead of this date.

2. With all the different types of desktop technologies out there now:

DATA CENTER BASED INFRASTRUCTURE
-Remote Desktop Session Host (Terminal Server)
-VDI (Citrix XenDesktop, VMware View, etc)

CLIENT BASED INFRASTRUCTURE
-Client VM – Hypervisor on baremetal client (Citrix XenClient, VMware View Local Mode, Virtual Computer NxTop, MokaFive, Virtual Bridges VERDE, etc)
-Client VM – Hypervisor on existing OS (VMware Workstation, Virtual PC, Fusion, Parallels, XP Mode, etc.)
-OS Streaming
-Traditional desktops

Companies are having a hard time trying to decide what is best for their users. One of the slides shown showed the enumeration of these technologies into the environment and I swear, it was like a pack of Skittles. There are so many directions to go and for many different reasons but you really have to factor everything or you are going to have end up with one of these Skittles environments over time. The biggest take away here is that there is no ONE solution that is the BEST. Your infrastructure might be a few of these. But don’t let it run away from you, plan it well from the get go. Pretty much common sense to anyone working with VDI already.

3. So my co-worker asked how do you plan your VDI environment? How do you know what is best? Brian recommended a couple of products that can help you:

-LakeSide Solutions VMP – pulls stats on your environment and spits out a report that tells you which users are good candidates for VDI vs. Terminal Server vs. staying on a traditional desktop.

http://www.lakesidesoftware.com/systrack_vmp.aspx

-Liquidware Labs – does a lot of the same and comes with 30 day trial

http://liquidwarelabs.com/products/stratuspherefit.asp

Another really good tool he suggested that was good for looking at the app compatibility and testing closer was App-DNA AppTitude. It does automated testing against 68,000 data points. They were acquired by Citrix just a couple of weeks ago:

http://www.app-dna.com/about-apptitude/automated-application-testing/

ChangeBase offers something similar and they were acquired by Quest just a few weeks ago:

http://www.changebase.com/

4. User workspace management. Let’s face it, any large VDI deployment across all the different infrastructure technologies above + across all the different OSes is going to be a nightmare if you don’t have some way of managing the user workspace. Brian talked about the concept of layering hardware, OS, application, and users. Not unfamiliar to anyone that has worked with VDI. He talked about 3rd parties like AppSense, RES, LiquidWare Labs, etc. vs. what’s being offered from Citrix, Microsoft, and VMware. I’m not going to go too much into this because I plan on writing an article about user workspace management and a comparison of the different options soon. But just know that this is a big point that will break or make your VDI project.

These were some of my main takeaways from the seminar without going into too much detail. There was a lot of information covered and I have to thank Brian & TechTarget for putting this on. Doesn’t matter if you’re just in the planning stages of your VDI project or if it’s been running great for years, I guarantee you will learn something new or what’s coming down the pipe in the VDI world at one of these seminars. You can sign up for one in your city here:

http://events.techtarget.com/desktopvirtualization/

Thoughts on Desktop Virtualization seminar with Brian Madden is a post from: JasonSamuel.com

More of my posts you might like:

1. Using Firefox in an environment with roaming profiles

1. Plug the USB drive to your XenServer and turn it on if it’s not on already.

2. Once it finishes booting, enter the Local Command Shell (type your root credentials)

3. Now type:

fdisk –l

this will list all the drives. Typically your USB drive will the best last one. Mine was /dev/sdb1 and under the System column, listed as HPFS/NTFS.

4. Now create a storage repository by typing:

xe sr-create name-label="USB drive" content-type=user type=lvm device-config:device=/dev/sdb1

When it completes, it will give you a UUID (long alpha numeric string). Don’t worry, no need to write it down.

5. Now open up XenCenter, and check the storage available on the XenServer you are working on. You should see one called “USB drive” or whatever you decided to name the SR. The UUID you saw above on the console will also be displayed if you click the storage repository in the resources pane and look under the General section. Click the screenshot below for a bigger version:

How to use an external USB drive as a storage repository in XenServer 5.6 is a post from: JasonSamuel.com

More of my posts you might like:

1. Running Open Solaris on Citrix XenServer 5.5 shows no IP address associated with the VM
2. How to move SQL error logs (ERRORLOG) to a different drive

Mount the image into the guest using:

mount /dev/xvdd /mnt

Then as the root user, execute the install script:

/mnt/Linux/install.sh

Then reboot the VM and it should work.

Running Open Solaris on Citrix XenServer 5.5 shows no IP address associated with the VM is a post from: JasonSamuel.com