JasonSamuel.com

When you try to start a VM in XenServer that talks to a StorageLink Gateway server, you get:

1/19/2012 x:xx:xx PM Error: Starting VM 'xxxxxx' - Storage assignment failed
(SSL_ERROR_SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
verify failed)

in the XenCenter log. You can verify the SSL cert by opening up the following in a browser window and replacing the x’s below with your StorageLink server’s IP address:

https://xxx.xxx.xxx.xxx:21605

You will get a cert error message in your browser. Notice the Citrix CVSM SSL certificate issued on 1/19/2009 has expired today 1/19/2012 at 20:25:53 PM (GMT) which is 2:25 PM Central Standard Time. So basically any VM you try to turn on, reboot, or migrate after the cert expired will not work and return the SSL error above in the XenCenter log. Yeah, big problem.

I was the first to call in about this issue soon after the cert expired apparently. As I was on the phone troubleshooting this with the support engineer, others began calling in with the same problem. We have escalated it to the highest level at Citrix support and have been assured a workaround and a new cert are both being worked on and something should be available tomorrow morning. This is going to impact pretty much all StorageLink customers globally so trust me, they are working on it. Over the past several hours, I have tried numerous workarounds myself but been unable to get a full fix yet. I’ve tried self signed certs using OpenSSL, IIS & SelfSSL, etc. but to no avail so far. The StorageLink Gateway does not use a web server such as lighttpd, Apache, Tomcat, etc. either so I can’t force it to use another set of certs on that end. Apparently it uses API calls. When you restart the services, you will notice it copies the following SSL certs which are the culprits (into memory I’m guessing). I used Process Monitor to verify:

D:\Program Files (x86)\Citrix\StorageLink\Server\cacert.pem

D:\Program Files (x86)\Citrix\StorageLink\Server\server.pem

I actually did manage to get a little further than I thought on the handful of workarounds I tried, but nothing completely successful yet to regain functionality while we wait for a hotfix. If you want to try playing with the certs yourself, just remember to restart the StorageLink services after you swap out the certs each time so it pulls them in. XenCenter should see the SSL cert change and prompt you almost immediately with a warning message.

I will keep this post updated with the latest developments. Please post if you are having the same issue or come up with a temporary fix. In the meantime, call Citrix and open a case so you are in the loop when the fix is released.

UPDATE January, 24th, 2012 – Citrix has published the fix:

http://support.citrix.com/article/CTX131994

Apply the certs using the instructions in the KB. Shouldn’t take long at all.

Today I would like to go over proper URL redirection when using SSL but first I would like to preface this by describing what proper URL redirection is. Anytime you inherit, consult for, or build an environment out from scratch, you must always verify every vector a user may use to get to your website. You must plan accordingly to intercept and redirect the user transparently when they manage to get to URLs that you don’t want them to use. They should never be prompted for an action (such as “click here” to continue) or receive a warning/error message if at all possible.

A 301 redirect is in my opinion to best way to steer a user to where they need to go. A 301 tells the browser/search engine spider that the redirect is permanent. In the case of spiders, a 301 will tell it only the destination URL should be indexed. A 301 redirect works fine over regular HTTP connections and all web servers and load balancers are capable of performing them. Alternatively, you can also use a 302 redirect but this tells spiders the redirect is temporary. Not very good for SEO purposes if your organization is concerned about that.

The problem occurs when you try to do a 301 or 302 redirect to an SSL URL (HTTPS URL) but the SSL certificate for that URL does not match the domain. It will throw a warning message saying the connection is untrusted or that there is a problem with the website’s security certificate in the user’s browser. Most users will come across on an HTTP connection such as http://www.domain.com and if needed, the network/web engineer would setup a web server or load balancer policy to perform the 301 redirect to https://www.domain.com. The certificate in this case is valid for “www.domain.com” and will not throw an error. But what if the user somehow manages to go directly https://domain.com over an SSL connection? It would throw an error unless the certificate is valid for “domain.com” which is most cases it is not. Even if you had a wildcard certificate for “*.domain.com”, it would not prevent you from getting a warning because this specifies any subdomain is valid, but does not cover just the domain by itself. Your user will always get a warning message. You may think you could write a policy on your load balancer to do the redirect but that will not work. The reason is a TCP connection has to be established first and during the SSL handshake before the connection is fully established is when the user gets the warning. The policy will never run as a result until after the user accepts the warning message and the SSL handshake is completed.

There are a couple of ways to get around this. The cleanest way of doing it is using a Subject Alternate Name (SAN) SSL certificate. A SAN cert allows you to specify exactly what domains you would like the certificate to be valid for. The price on SAN certs is much higher than a standard certificate or even a wildcard certificate and they are only available from a handful of certificate authorities at this time. They require a lot more validation typically and the validation term is usually a bit lower than a standard certificate.

Let’s compare a SAN SSL certificate vs. a Wildcard SSL certificate. A single SAN cert would cover me for the following if I wanted it to:

domain.com
www.domain.com
www.myotherdomain.com

while a single Wilcard SSL cert would only be able to do the following:

*.domain.com

It is possible to combine both so you could get a SAN + Wildcard SSL certificate that will cover all of the above. To my knowledge, the only certificate authority that currently offers them is DigiCert. They have a pretty nice write up about their cert here:

http://www.digicert.com/ssl-support/wildcard-san-names.htm

I once asked a VeriSign managed PKI rep why they don’t offer wildcard + SAN like DigiCert and he went on for 5 minutes about ICAAN policy. Why one cert authority can issue while another won’t, I am guessing it likely has to do with acquisition costs for them. You have to remember, VeriSign is still trying to convince people to buy SGC SSL certs (for legacy browsers) at a premium price when in my opinion, SGC has been obsolete for years now and any web/network engineer that knows about SSL encryption has anything below 128 bit locked down anyhow. But I guess if people will buy it, they will sell it. Either there just hasn’t been enough demand or their costs are too high for wildcard + SAN certs. Not trying to pick on VeriSign, just making an observation. I still purchase certs from them regularly (non-SGC of course). Most every large organization does, but sometimes the wildcard + SAN might be the best solution for you. I know some CDN providers like Akamai sometimes have a need for them and purchase through DigiCert.

Getting back on subject, a second way of getting around this is purchasing two standard SSL certifcates and putting them on different IPs. One will do a 301 redirect to an another. So you would have to purchase an SSL cert for domain.com AND www.domain.com.

Let’s look at some real world examples to better understand this. I am going to compare Chase, Bank of America, Citrix, and Amazon.com to show the different types of solutions for this problem.

Chase.com
Chase uses SAN certs from VeriSign. Again, SAN certs are a very pricey approach but the easiest. Your configuration on your load balancer and servers is minimal since one SSL cert will take care of everything. This will result is less overhead on your load balancers as well which is a big plus when you serve up millions of connections.

Go to https://chase.com, it will immediately do a 301 redirect to https://www.chase.com. Here is a screenshot showing the 301 redirect to https://www.chase.com and the certificate which shows the 4 Subject Alt Names Chase’s cert is valid for:

As you can see, the certificate is valid for www.chase.com, chase.com, www.yourchasefreed.com, and yourchasefreedom.com. You will also notice that the cert is only valid for a 1 yr period. Typically CAs will try and limit you to a max of 2 years on SAN SSL certs and EV SSL certs but it is always best practice to keep them to 1 year just in case your cert keys were compromised. A SAN cert can have many domains leaving you very exposed, so it is a good idea to change it every year for this reason.

Bankofamerica.com
Bank of America uses standard SSL certs from VeriSign. Cheaper but a little extra configuration involved on the load balancer. Let’s take a look at what happens when you visit https://bankofamerica.com:

You will be 301 redirected to https://www.bankofamerica.com just as you were with Chase. But this time using a standard SSL cert with one CN name as you saw above. You can verify this by pinging bankofamerica.com which resolves to 171.159.100.173 and typing just the IP into the browser over SSL as https://171.159.100.173. This will give you an SSL error before the 301 redirect gets a chance to run and you can view the cert:

So just to be clear, typing https://bankofamerica.com will use the above cert which is valid, will complete the connection, run a policy that will do a 301 redirect to https://www.bankofamerica.com, cert for https://www.bankofamerica.com is validated, then the connection is completed and the page is served up. A perfectly good way around the problem using standard SSL certs as opposed to SAN SSL certs.

Citrix.com
Citrix is another example of using the standard SSL certificate redirect method much like Bank of America. But they do not leverage the 301 redirect at all. Instead they use a 302 temporary redirect pointed directly to their index page. The 302 is likely done for SEO purposes so the “www.citrix.com” domain is indexed in search results and not the “www.citrix.com/lang/English/home.asp” destination page. In this case, a Netscaler with either a responder policy or content switching policy will send you to https://www.citrix.com or https://citrix.com which both display the same content. They are on different VIPs with different SSL certs. One is a standard cert while the other a wildcard. They could have used two standard certs if they wanted to as well of course but in this case, they may use the wildcard for other subdomains, GSLB, etc. and the standard cert is meant to solely capture those users typing in an incorrect URL:

Amazon.com
Woops! The world’s largest online retailer does not seem to be redirecting customers that arrive on an irregular URL over SSL properly. If you go to https://amazon.com, it will give you a “This Connection is Untrusted” error. It’s not the end of the world but still, a user should never see a message like this.

In IE for example, it will say “There is a problem with this website’s security certificate”. Again, not the end of the world. But in Chrome, the error will have a bright red background and say “This is probably not the site you are looking for!
You attempted to reach amazon.com, but instead you actually reached a server identifying itself as www.amazon.com. This may be caused by a misconfiguration on the server or by something more serious. An attacker on your network could be trying to get you to visit a fake (and potentially harmful) version of amazon.com. You should not proceed.”

Yikes! Do you really want your users to see a message like this?

Interesting to note however, going to https://www.amazon.com does a 301 redirect to http://www.amazon.com. Obviously Amazon wants the customer to use HTTP rather than SSL on their homepage. Much less overheard on their load balancers plus they don’t have a login box directly on their home page so no need for SSL to begin with. This is thought out well but I am puzzled as to why they don’t do anything about https://amazon.com. As you saw with Chase, Bank of America, and Citrix, the different solutions are very easy to implement.

I hope this article has been helpful. Please post a reply if you have a different solution to the problem that I did not cover. I would love to hear your insights.

There are 4 easy ways to check that SSLv2 and weak ciphers are disabled on your web servers and appliances. SSLv2 is deprecated and should never be used. You will definitely need to verify these are disabled for PCI compliance and SOX compliance. The instructions below apply to check SSLv2 on any web server (IIS, Apache, iPlanet, etc.) or any traffic or load balancing appliance (Citrix Netscaler, F5 BigIP, etc.).

1. SSLDigger - Download it here http://www.foundstone.com/us/resources/proddesc/ssldigger.htm. It’s a great little tool that first tests for SSLv2, then gives you the option to test for weak ciphers one by one, then spits our a nice little HTML report of the findings. You can even create a .txt file with all your URLs to test and perform testing in bulk. This is wonderful when you have many domains to test and need to give a report to management quickly.

Checking the Google.com SSL cert (click to enlarge):

Checking for weak ciphers systematically (click to enlarge):

HTML report (click to enlarge):

2. OpenSSL – The old favorite. Type the following to connect to your web server or appliance over SSLv2:

openssl s_client -connect www.yourdomain.com:443 -ssl2

It should connect giving you a CONNECTED(000007C0) message but then immediately fail and display an error message like one of the following:

419:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:
420:error:1406D0B8:SSL routines:GET_SERVER_HELLO:no cipher list:s2_clnt.c:450:
10076:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:. \ssl\s2_pkt.c:428:
write:errno=548965:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:
write:errno=10054

SSL handshake failures and connection resets (errno=10054) are completely normal if SSLv2 is disabled. The specific error message you get will depend on exactly what behavior your web server or appliance is configured to do when a client connects using it.

Example of SSLv2 disabled:

Example of SSLv2 enabled (VERY VERY BAD!):

3. Nessus – Download it here http://www.nessus.org and use the Home Feed and setup a new scan using the SSL Anonymous Cipher Suites Supported and SSL Version 2 (v2) Protocol Detection plugins. Please make sure to disable all the other plugins, you don’t want all that hitting your server. Just these 2 are sufficient. Then check the Reports and see if it is reporting SSLv2 as a vulnerability.

4. External SSL checkers – Running nice little tools like the above three is nice to check your servers when you are internal but if you want to see what 3rd parties are seeing externally, use one of these online SSL checkers to test:

http://foundeo.com/products/iis-weak-ssl-ciphers/test.cfm
http://www.dnsstuff.com/tools/

I’ve covered how to convert an SSL cert to a .pem with a private key for a Netscaler before here but sometimes you might be required to install a full SSL cert chain (read my article here on how SSL certificate chains work). I actually install the full SSL chain (root and intermediate certs) on any Netscaler I deploy as a best practice.

I’m going to use Network Solutions again in this example but this will work for Verisign, GeoTrust, Thawte, Comodo, or any other CA (certificate authority).

Here is how you do it on the Netscaler:

1. I’m going to assume you have already generated a CSR (either through the appliance or IIS, whatever your preference), had your cert validated, and have downloaded your site cert and intermediate/root certs from Network Solutions since I have covered this before in the articles I mentioned above.

2. Now just login to your Netscaler, install your website cert that was issued to you first under SSL > Certificates on the Netscaler.

3. At this point, if your website is publicly available, let’s examine the cert chain. Go to http://www.sslshopper.com/ssl-checker.html and type in your website (if it’s not publicly available, use OpenSSL which I have written about here). Since you have not installed intermediate certs on your Netscaler yet, it should report your SSL cert is good but your SSL cert chain is broken like below:

4. Now go back to your Netscaler and install the two intermediate certs and one root cert that Network Solutions has give you. You really only need the intermediate cert but I like to install them all just in case. They are in .crt format when you get them from Network Solutions and that is fine, no need to convert. Just go to SSL > Certificates on the Netscaler and click Add in the bottom toolbar. Give it a name and use Browse (Local) to find your certs and install all 3 one by one. Here is what it should look like when you are done:

5. Now here comes the most important part. You need to link the intermediate cert (NetworkSolutions_CA) to your site’s SSL cert. Right click on your site’s SSL cert and hit “Link”. Now choose the NetworkSolutions_CA cert in the drop down box and press OK. It will confirm the cert is now linked. The Netscaler actually supports up to a 10 cert chain so you can have your website cert + 9 more intermediate certs linked to it if you want. So feel free to link more if you need to at this point. You can also click on the “Cert Links” button in the bottom toolbar and it will show you all your links. For example, here is a Netscaler with 3 certs from Network Solutions that have been linked to the “NetworkSolutions_CA” cert:

6. Now go back to http://www.sslshopper.com/ssl-checker.html and try your domain again. This time it will say your SSL chain is good:

Here’s an example where I linked each intermediate cert chain all the way down to the root certificate. I would recommend you link all your certs down to the root as a best practice:

Also, Citrix put together this really good guide on SSL cert chains I am hosting here for you. It’s a great read with plenty of screenshots so take a look:

Citrix_Verisign_NetScaler_SSL_Cert_Deployment_Guide

Cisco WLC (Wireless LAN Controllers) require the entire SSL cert chain to be installed on the appliance. I like to do this via IIS since it is very quick and efficient and all my intermediate and root certs are already on there typically (double check your servers please).

1. Create a temporary site in IIS and generate a CSR. Make sure the domain matches whatever appliance you are trying to generate it for exmp: wireless.yourcompany.com.

If you are exporting a wildcard cert that might already be on your IIS server, just export it as a PFX and skip to step 5.

2. Send the CSR to your registrar and wait for them to validate.

3. Get the .crt and convert it to .cer (just open the .crt and save it as .cer). Do the same for the intermediate and root certs you received.

4. Install the .cert in IIS as well as all intermediate and root certs they send you.

5. “View” the cert in IIS (Site properties > Directory Security > View Certificate > Details Tab > Copy to File) and export it with the private key in pkcs12 (.pfx) format. Also make sure to check “Include all certificates in the certification path is possible”. Just set the password as “password” so it’s easy to remember.

6. Now you have a .pfx with your whole SSL cert chain but you need to convert it to .pem for the WLC.

7. Use this command with OpenSSL:

openssl pkcs12 -in yourcert.pfx -out yourcert.pem

8. Type the “password” when prompted

9. TFTP the new .pem containing the whole SSL cert chain and private key over to the WLC and reboot. You will have to specify the “password” when you type in your TFTP info and Save and Reboot the WLC after the download is successful. Test it out and it should work.

Often times you might have to add an SSL cert to a Cisco or Citrix appliance (like a Netscaler) but it only accepts the SSL cert and private key in a certain format.  Your CA (certificate authority) will issue the SSL cert to you as a .crt.  From there, you need to package the cert with the private key and in the format your appliance requires.  There are different ways to do this and one easy way I like is using IIS and OpenSSL which is a free open source tool and can be downloaded here:  http://www.openssl.org

1. Create a temporary site in IIS and generate a CSR.  Make sure the CN (common name) you enter matches whatever appliance you are trying to generate it for exmp:  wireless.mycompany.com

2.  Submit the CSR to your CA such as Verisign, GoDaddy, Network Solutions, etc. and wait for them to validate and issue your cert.

3. Get the .crt from the CA once they issue it and convert it to .cer.  Just open the .crt by double clicking on it and and save it as .cer using the export wizard.  This is necessary since IIS does not accept .crt.

4. Install the .cer in IIS using the Web Server Certificate Wizard where you originally generated the CSR.  Just process the pending request to install it.

5. View the cert in IIS and export it with the private key in pkcs12 (.pfx) format.  Just set the password as “password” since you will be deleting it after conversion is complete later.

6. Go to the directory where OpenSSL.exe is and type in the following to convert the .pfx to a .pem

openssl pkcs12 -in nameofcert.pfx -out nameofcert.pem

7. Type the “password” when prompted for the pass phrase.

8. Install the .pem on the appliance and it should work

IMPORTANT NOTE:

If you are doing it for some appliances like a Cisco IronPort, you need to add the nodes switch when creating the .pem:

openssl pkcs12 -in nameofcert.pfx -out nameofcert.pem –nodes

The –nodes switch ensures that the key inside the .pem is left unencrypted.  If you attempt to install a .pem created without the -nodes switch, the appliance will take the cert but will not accept the private key since it cannot read it in an encrypted state.