JasonSamuel.com

By default, Firefox stores it’s profile cache in "C:\Documents and Settings\yourusername\Application Data\Mozilla\Firefox\Profiles". This can be problematic when using roaming profiles with a limited amount of profile storage space due to quotas. As you add extensions, it grows even bigger. This is a big problem in XenApp/XenDesktop/View and other VDI environments using roaming profiles. You’ll soon start getting errors saying you are almost out of profile storage space or that you have exceeded your profile storage space. Something like this (note I just grabbed this screenshot from a Google Image Search, you would normally see a bunch of Mozilla Firefox related files in there eating up the space):

The solution is to move the profile out of the user folder and into another local folder or onto a network drive/home folder, thus reclaiming the profile storage space. For example, I created "c:\FireFoxProfile" to house the profile. Here is what you need to do to begin using this folder:

1. Edit "C:\Documents and Settings\yourusername\Application Data\Mozilla\Firefox\profiles.ini" in Notepad and change:

IsRelative=1

to

IsRelative=0

What this does is allow you to use an explicit path that you specify for the profile.

2. Change the next line from:

Path=Profiles/xxxxxxx.default

to

Path=c:\FireFoxProfile\xxxxxxx.default

3. Save profile.ini and restart Firefox. You will notice that immediately, the new profile folder is populated. Keep in mind when you open Firefox for the first time, it will be like a brand new install and you will lose all of your bookmarks, extensions, customizations, etc. unless you restore them from the old path.

4. You can delete everything in your old profile path once you have Firefox working in the new profile path. This will immediately reclaim your roaming profile storage space.

NOTE: If deploying to the entire corporation, there are other methods to centrally manage Firefox and control the deployment. This solution is good for the one off users though.

I’m building out a quick test lab environment at home tonight using XenServer 5.6 SP2. Unfortunately the server I happened to be using for this is a 1U box with only two 73 GB drives inside. El cheapo Dell. Even striped, I needed just a bit more storage for what I was intending to use it for. Luckily I had a 250 GB external Western Digital USB drive handy. It was formatted as NTFS since I pulled it from a Windows desktop. I figured why not use it as a local storage repository? It’s a temporary test lab after all.

1. Plug the USB drive to your XenServer and turn it on if it’s not on already.

2. Once it finishes booting, enter the Local Command Shell (type your root credentials)

3. Now type:

fdisk –l

this will list all the drives. Typically your USB drive will the best last one. Mine was /dev/sdb1 and under the System column, listed as HPFS/NTFS.

4. Now create a storage repository by typing:

xe sr-create name-label="USB drive" content-type=user type=lvm device-config:device=/dev/sdb1

When it completes, it will give you a UUID (long alpha numeric string). Don’t worry, no need to write it down.

5. Now open up XenCenter, and check the storage available on the XenServer you are working on. You should see one called “USB drive” or whatever you decided to name the SR. The UUID you saw above on the console will also be displayed if you click the storage repository in the resources pane and look under the General section. Click the screenshot below for a bigger version:

I’ve had this happen several times when an ESX host is stuck thinking it is going into maintenance mode from previous maintenance. You get an error saying it cannot complete the migration and “The operation is not allowed in the current state”. Very cryptic error. Just restart the management agents on the ESX host by SSHing in and typing:

service mgmt-vmware restart

hit enter, then:

service vmware-vpxa restart

and hit enter again. You should see the services restart with an "[ OK ]" after each line.

Try to vMotion again now. If you still get an error, disconnect the ESX host from Virtual Center and then reconnect it. Try to vMotion again and it should work. You can also try restarting VC services too. If you have a better solution, let me know. I know it is a bit tedious to go through this process but it does work.

I came across an interesting issue trying to setup an additonal web server with Solarwinds NPM. Everytime I tried to download templates from Thwack, it have give me an “Unable to contact thwack server” error. This happens sometimes when using a proxy. The fix is adding this to your C:\Inetpub\SolarWinds\web.config:

<system.net>
  <defaultproxy enabled="false"></defaultproxy>
</system.net>

You don’t have to reset IIS or restart any Solarwinds services. Just refresh the page and you should be able to see Thwack templates now.

PowerPath/VE replaces the native mulipathing plugin which gives you incredible performance gains when managing your EMC Clariion SANs. You have to push this plugin to all your vSphere/ESX hosts to take advantage of it. Using VMware Update Manager is the easiest solution but I ran into a problem. Adding a 3rd party extension requires adding a patch repository in VUM and it only accepts web URLs. So I would have to setup a dedicated web server just for this patch repository which is ridiculous. Instead I decided to leverage my Virtual Center server itself since I knew it uses a Tomcat installation to display the vCenter web GUI and webservices. Some quick Googling and I found Joshua Townsend had the exact same idea and wrote a nice guide to do this already:

http://vmtoday.com/2010/02/installing-powerpathve-using-vmware-update-manager/

Once I set it up the way Joshua describes and pointed VUM to the patch repository URL hosted by vCenter, it worked perfectly. You will notice in my screenshot below Connectivity Status says “Connected” now.

Masking the web server software in your http header/http server banner (server header obfuscation) is an important layer of security you might want to implement since it can be accomplished so easily on a Netscaler. It’s a layer of security you can add to prevent someone from telling what kind of web server you are running, though it is still quite possible to figure out depending on your environment and application so don’t rely on it too much. The thought is an attacker scans for certain versions of a web server that have known vulnerabilities and begins running attacks for that specific web server software to see if those vulnerabilities have been patched or not. They can do this programmatically so changing the header to say something else is a layer of security that can prevent them from easily figuring out what your web environment infrastructure is like.

You will notice that Chase.com uses “JPMC1.0″:

Amazon.com uses just “Server”:

Google.com uses “gws”:

This can be done very easily using rewrite policies on the Netscaler. Catherine Hampton wrote a great article over at the Ctirix Developer Network on how to do this:

http://community.citrix.com/display/ns/Using+Rewrite+to+Improve+Web+Server+Security

And if you want to read more about web server fingerprinting, check Net-square’s website and their httprint tool:

http://net-square.com/httprint/

Saumil Shah at Net-square wrote an excellent and very thorough article on HTTP fingerprinting here:

http://net-square.com/httprint/httprint_paper.html

Bad cell phone signal reception is something everyone has to deal with, both home users and business users. There is a lot of conflicting information out there and I often have to explain how cell phone repeaters and other cell phone booster solutions work and what doesn’t to my family, friends, and co-workers. I have very good first hand experience with some solutions and since it is technology related, I have decided to dedicate a post here on my blog about these cell phone reception solutions to point everyone to. Saves me from having to say the same thing over and over when I could just email them a link.

First, for those people that are not regular readers of my blog, I have to let you know I work in IT infrastructure so cell phone engineering is not my expertise. I do some telecom and VoIP work but I am not an RF Engineer that works for a cell phone carrier by any means so please don’t consider me an expert on this subject. I’m only going to tell you about my own experience and research.

Second, I live in the Houston, Texas area. Cell phone coverage depends on your carrier and how their cell tower coverage is in your city. Houston being the fourth largest city in the United States has very good cell tower coverage from all the big carriers: AT&T, T-mobile, Verizon, Sprint, & Nextel. So parts of this post may or may not apply to you depending on where you live. Please do your research before spending your money and don’t rely soley on my blog post.

The Cause of Bad Reception
The question I get asked a lot is how can I improve the cell phone signal in my house or my office because I get 0 bars. In Houston, it is rarely a cell phone tower coverage issue. Cell phone towers are everywhere in this city. People just usually don’t notice them. But if you really start paying attention, you can’t go more than a few blocks before spotting one. Most of the time I hear complaints from friends and family that have just moved into a new house (newly constructed home) about cell reception being poor or 0 bars. They wonder what the cause could be in their brand new house because their old house didn’t have signal issues. The reason for this is because new homes use better radiant barriers in the attic than older homes. Especially in hot places like Texas. These radiant barriers are the cause of the poor signal, not cell phone tower coverage. They are blocking the cell phone signal from penetrating the house. Almost all home builders in Houston use LP TechShield radiant barrier:

http://www.lpcorp.com/techshield/

David Weekly Homes, Newmark Homes, etc. to even smaller custom home builders all use it here in Houston. TechShield can save you a TON of money per year. I went from having a $900 per month electricity bill in the summer in a house with no TechShield to less than $200 in a newly constructed home with TechShield installed. And the new house has double the square footage! Though TechShield works great and will save you a lot of money on your energy bill, the downside is it block RF signals (radio frequency signals). RF signals is how your cell phone talks to the cell tower. Specifically Ultra High Frequency or UHF. So what a lot of my friends, family, and co-workers find is that when they go home, they go from having 5 bars outside to 0 bars inside their homes. You can confirm TechShield is the cause by going into your attic and looking up. You will see the silver radiant barrier and it will say TechShield on it like this:

With businesses, the taller your building is and the more steel, insulation, and other dense material it is constructed with, the less signal you will get. In some cases like hospitals and large medical centers there could be equipment causing poor signal too. A lot of the big skyscrapers in Downtown Houston and the Medical Center rarely get good a good cell phone signal without working with carriers to install commercial in-building cell phone repeaters on every floor. I will go over what cell phone repeaters are exactly in the next section.

The Wrong Solution
A quick Google search will tell you cell phone boosters or cell phone repeaters are the way to go to solve bad cell phone signal issues. Those stickers and other solutions that are “passive” are all garbage. You need something that is “active” and amplifies the signal which is what a cell phone repeater is. Another term for a cell phone repeater is a BDA or bi-directional amplifier. For home users, you can even buy cell phone signal repeaters at electronic stores like Fry’s or Best Buy. They are cell phone amplifiers that amplify the weak signal from cell phone towers. Usually you stick an antennae on your roof that pulls the signal in, which is connected to the amplifier inside your home which amplifies the signal, which is then connected to another antennae inside your home that your cell phone connects to. Here is a diagram of how a simple cell phone repeater system works:

One thing you will not be told about these is that though they are completely legal to purchase, they are illegal to operate. The reason is cell phone carriers pay tons and tons of money to the FCC to license radio frequency spectrums (cellular frequency spectrums) for their use. You are not allowed to broadcast on these licensed frequencies without permission from the carrier that has licensed the spectrum or you will be fined by the FCC. The problem with a lot of cell phone repeaters is that they amplify the signal, sometimes way too much. This can cause issues with a nearby cell phone tower and causes problem for the carrier. For this reason, they do not want consumers installing and operating in-building cell phone repeaters in their homes. I speak from first hand experience so let me continue.

Amazon, Best Buy, Fry’s and a lot of other stores and websites sell dual band cell phone repeaters and boosters. They are usually dual band meaning it will work with 800 Mhz and 1900 Mhz bands which is AT&T and T-mobile respectively in the US. My friend’s 2 story 5,500 sq. ft house was currently getting 0 bars inside the house due to TechShield radiant barrier. Up to the front door, we could get 5 bars on AT&T and T-mobile phones. As soon as we stepped into the house, the signal drops. He specifically wanted coverage in the kitchen and living rooms only because that is where most of the family usually is. Having 0 bars in the house was unacceptable. If there was an emergency at work or with family everyone has his cell phone number and will call it first. Also he did not have a land line since his whole family had cell phones. So he purchased the zBoost YX-545 unit for around $270 from Amazon.com.

http://www.amazon.com/Wireless-Extenders-YX545-Dual-Band-Booster/dp/B003VOW5WI/ref=sr_1_1?s=wireless&ie=UTF8&qid=1301611766&sr=1-1

You will see a lot of reviews online ranging on this unit and it’s predecessor the YX-510 from horrible reviews to 5 stars. It all depends on your building, cell tower coverage, and installation. I personally saw very poor results when my friend set it up. Within 5 ft. of the unit, I went from 0 bars to 2 bars. As soon as I stepped out of this 5 ft radius, I got 0 bars again. He called their support line which was very helpful and they confirmed everything about the install was correct. He even sent them pictures of how the antennae was mounted outside on the roof and into the house. But, that’s the best he could get in the house.

So both of us being geeks, I convinced him to try upgrading the unit. He bought the premium kit with upgraded outdoor directional antennaes for $130. You can use this to get a better signal from the cell tower because you can point it directly at it:

http://www.amazon.com/Wireless-Extenders-zBoost-Dual-Band-Directional/dp/B003N3HDAQ/ref=sr_1_5?s=wireless&ie=UTF8&qid=1301611766&sr=1-5

We also upgraded the thin RG-59 coaxial cable the kit came with to some nice thick RG-11 cable:

http://www.showmecables.com/viewItem.asp?idProduct=7326

The tradeoff is the RG-11 is less flexible so it is harder to run the cable through walls or attic all the way up to the roof. Imagine RG-6 (the cable you use to connect your to your TV or cable box for cable TV) but twice as thick.

After getting all the new upgrades installed, we could get 3 bars within about 10 ft. As soon as we leave that radius, the signal drops to 0. At this point zBoost support did not have any other suggestions for him except to move the directional antennae on the roof and point it directly at another cell tower. We had already mapped out where the AT&T and T-mobile cell towers were by using Google Maps and a cell tower locator application for our phones called “Cell Tower Locator”. We also drove by the cell tower sites to confirm we were connected to it and measured the dbm as we approached and left the tower. There are also websites online that document cell tower locations and we confirmed the locations of the towers there as well, it was only 1.3 miles away. He even rotated the antennae on a mast at 5 degrees each for the full 360 degrees testing and that was the best he could do. So it wasn’t a problem with my friend’s antennae pointing for sure.

So again, he opted to upgrade but this time to a business solution that is used for large office buildings, warehouses, etc. One really popular cell phone repeater company that caters to home users and business users is Wilson Electronics but he opted to go with Cellphone-Mate that makes a comparable product with excellent reviews. There was a company locally that installed both Wilson and Cellphone-Mate in office buildings and they recommended going with Cellphone-Mate for his house. He decided to go with their flagship product, the SureCall CM2020 68dB amplifier for $1200. They claim to cover 60,000 sq ft on their website:

http://www.cellphone-mate.com/newp/68db.html

and that it is FCC approved to the max output of 3 watts. But keep in mind they are talking about the output of the device being legal per FCC regulation. They do not say operating the amplifier itself is FCC approved and as I mentioned above, it is illegal to operate an unauthorized cell phone repeater/amplifier. He did not know this of course when purchasing the unit for $1200. They told him on the phone as well as on their website that their basic CM2020 kit can cover up to 10,000 square feet easily and they have done many installs for businesses and hospitals. Up to 60,000 sq ft. that the manufacturer’s website said was with multiple antennae upgrades and such that was overkill for a home. He purchased a complete top of the line kit consisting of the following:

-68 dB CM2020 amplifier
-Outdoor directional yagi antennae
-Indoor omnidirectional dome antennae
-LMR400 cables (really thick coaxial cable)

You can see it here:

http://www.wpsantennas.com/CM2020-Kit-Cellphone-Mate-65db-dual-band-system.aspx

My friend purchased it locally and got the company to come out to install it. Instantly his 5,500 sq ft. house was at 5 bars. Anywhere in the house was a solid 5 bars, no drops at all. The company tuned down the 68 db gain using the dip switches on the front of the amplifier to a level that was just enough to cover the inside of the house only.

Now the problem came a few days later when he received a certified letter from an RF engineer from one of the major cell phone carriers. In the letter the RF engineer stated that since the day the unit was installed, it has been causing interference with a nearby cell tower. It was causing all sorts of problems for that carrier’s customers in the area. So they had sent this RF engineer out to investigate. Using a directional antennae in his van and some other hardware, he discovered the source of the signal was my friend’s house which is why he sent the letter. He went on to explain cell phone repeaters (aka BDAs) are not permitted to be installed for use on any cell phone spectrum in the US by the FCC without consent from the carrier that licenses (T-mobile, AT&T, Verizon, etc.). None of these carriers permit the use of a BDA by a home or business user. Only the carrier themselves installs them and it’s a huge process and very expensive to have them do it so only large corporations tend to do this. My friend called the RF engineer and got more info on this. They typically install licensed BDAs themselves or subcontract the work out to companies like the one that my friend had hired to install the unit.

In fact a few months ago while working in one of the largest skyscrapers in Downtown Houston, I ran into a T-mobile subcontractor that was going from floor to floor testing the building’s repeater signal. They had several of them per floor that fed into an IDF closet that ran throughout the building and into several amplifiers on one level. Something like this:

Anyhow, operating a big system like above or a simple system like my friend started off within his house on your own is illegal. Only the carrier which is the licensee of the spectrum is allowed to do it. Quick research online shows several cases of FCC inspectors imposing fines and seizing cell phone repeaters that were installed by home and business users without authorization. They typically start of with cease and desist letters like below:

http://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0927/DOC-301700A1.pdf
http://www.fcc.gov/eb/FieldNotices/2003/DOC-300634A1.html
http://www.fcc.gov/eb/FieldNotices/2003/DOC-296857A1.html
http://www.fcc.gov/eb/FieldNotices/2003/DOC-296238A1.html
http://www.fcc.gov/eb/FieldNotices/2003/DOC-295061A1.html
http://www.fcc.gov/Daily_Releases/Daily_Business/2010/db1012/DOC-302031A1.txt
http://www.fcc.gov/eb/FieldNotices/2003/DOC-266448A1.html
http://www.fcc.gov/Daily_Releases/Daily_Business/2010/db1101/DOC-302541A1.pdf

Here is the important part in all these cease and desist notices:
“Licensees may install in-building radiation systems without applying for authorization or notifying the FCC, provided that the locations of the in-building radiation systems are within the protected service area of the licensee’s authorized transmitter(s) on the same channel or channel block.”2 A licensee’s authority to install a BDA does not permit a subscriber to install a BDA, unless that subscriber has received explicit authorization from the licensee to do so. In response to an inquiry from an FCC agent, T-mobile reported that it did not provide you authorization to install a BDA. Operation of radio transmitting equipment without a valid FCC authorization or license is a violation of Section 301 of the Communications Act of 1934, as amended,3 and may subject the responsible parties to substantial monetary forfeitures, in rem arrest action against the offending radio equipment, and criminal sanctions including imprisonment.”

These “notices” from the FCC go on and one if you do a Google search for “FCC BDA notice”. I’ve found them from just a few weeks ago to even as far back as 10 years ago. You will notice that most of the complaints above are to home or business users. Some are even found on boats. I even found one to Sony Pictures Studios in Los Angeles so even a huge company can land in hot water. One really interesting find, there are some notices from the FCC stemming from carrier to carrier complaints! So even one carrier installing a repeater that interferes with another carrier’s signal is a real issue and the FCC has to step in to correct it.

Luckily, the RF engineer was extremely nice in informing my friend of the laws in his letter and on the phone and the company that installed the unit had a 30 day return policy and removed the unit within a day. Apparently, this happens a lot in the Houston area. Consumers go and purchase cell repeaters (or BDA which is the “official” term) from Amazon or Best Buy and install them. Then he has the headache of hunting them down and requesting the business or home user to remove it. If they don’t, he then requests help from the FCC and they said a notice to the user like above. Poor guy. Working in infrastructure myself, imagine end users running amok on your network causing it to go down every day. This is what he had to deal with on a daily basis. I did not envy him.

The Right Solution
So now my friend was stuck. After spending 2 months experimenting with cell phone repeaters and finding out their operation was not allowed, he was back at square one. So I ended up doing research for him and found another solution called a femtocell I pointed him toward:

http://en.wikipedia.org/wiki/Femtocell

It is basically a small device about the size of your router that sits in your home. Your cell phone will connect to the device instead of a cell tower. The device then connects to your carrier’s network through the Internet using a secure tunnel. So you will be making the phone call over the Internet and not rely on a cell tower anymore. Currently AT&T, Sprint, and Verizon all use this technology. Since his family was on an AT&T family plan, he purchased their solution called the AT&T MicroCell:

http://www.wireless.att.com/learn/why/3gmicrocell/

The device itself is manufactured by Cisco so you know it’s a solid piece of networking equipment. It cost a one time fee of $150 and there were no charges after that. Plus AT&T gave him a 30 day trial. I was there the day he got it in and after following the simple setup procedures, we turned it on. It took a about 5 minutes for it to get a GPS signal (for Enhanced 911 purposes) and to establish the VPN tunnel with AT&T’s server. Within a few seconds of all lights going solid green, all the AT&T phones that were registered to use the device had 5 bars inside the whole 5,500 sq ft. house! A $150 femtocell unit was doing the same thing a $1200 cell phone repeater was doing! Both of us were floored. All that time and money he spent on installing was a complete waste. He’s been using the AT&T Microcell for about 8 months now and is extremely happy with it.

One thing to note, I mentioned earlier in this post that dual band repeaters work on two spectrums. But sometimes carriers that offer high speed data operate those services on a different spectrum that is not amplified by the repeater so you will get little or no high speed data when you are around a repeater. For example, T-mobile uses 1900mhz for voice and slow EDGE data service in the US but 1700/2100 MHz for high speed 3G data service. When you go with a femtocell from the carrier that doesn’t rely on cell towers, you don’t have to worry about frequency bands and data, everything goes through the Internet and you will get the high speed data service you pay for!

By the way I mentioned above every major carrier has femtocell except T-mobile. I read an article that they were testing them though. For now, look into T-mobile’s feature called “Wi-Fi calling” which is similar to femtocell but this requires you have a special phone that can make calls over WiFi. Read about it below:

http://support.t-mobile.com/doc/tm24195.xml

Go to :

http://www.t-mobile.com/shop/phones/default.aspx

and click the “Wi-Fi and Mobile Calling” checkbox in the left hand bar to see all the phone that have this feature.

Conclusion
Do your research before investing money. My friend and I are both geeks, and he makes a lot more money than I do so he doesn’t care about throwing money away experimenting. If you live in a rural area with no cell phone towers around, using an unauthorized BDA or cell repeater will likely not mess with a carriers’ network and you probably won’t get a letter from a carrier or the FCC. But if you live in a highly populated city like Houston with cell towers everywhere, the chances of your repeater causing issues is much higher. Cell phone repeater companies will say “FCC approved” on the device but again as I mentioned earlier in the post, they are talking about the device itself, not it’s operation by you. So it is risky to use one and I would personally never attempt it.

Luckily for us, femtocell technology is available from most carriers and works just as well as an enterprise level cell repeater and for a fraction of the cost. So now my friends, family, co-works can read this article and I don’t have to keep repeating my story over and over again. And everyone else reading this, please do post if my experience helps you any. I’d also really love to hear from any RF engineers that work for any carriers about their own experiences and recommendations.

Today I would like to go over proper URL redirection when using SSL but first I would like to preface this by describing what proper URL redirection is. Anytime you inherit, consult for, or build an environment out from scratch, you must always verify every vector a user may use to get to your website. You must plan accordingly to intercept and redirect the user transparently when they manage to get to URLs that you don’t want them to use. They should never be prompted for an action (such as “click here” to continue) or receive a warning/error message if at all possible.

A 301 redirect is in my opinion to best way to steer a user to where they need to go. A 301 tells the browser/search engine spider that the redirect is permanent. In the case of spiders, a 301 will tell it only the destination URL should be indexed. A 301 redirect works fine over regular HTTP connections and all web servers and load balancers are capable of performing them. Alternatively, you can also use a 302 redirect but this tells spiders the redirect is temporary. Not very good for SEO purposes if your organization is concerned about that.

The problem occurs when you try to do a 301 or 302 redirect to an SSL URL (HTTPS URL) but the SSL certificate for that URL does not match the domain. It will throw a warning message saying the connection is untrusted or that there is a problem with the website’s security certificate in the user’s browser. Most users will come across on an HTTP connection such as http://www.domain.com and if needed, the network/web engineer would setup a web server or load balancer policy to perform the 301 redirect to https://www.domain.com. The certificate in this case is valid for “www.domain.com” and will not throw an error. But what if the user somehow manages to go directly https://domain.com over an SSL connection? It would throw an error unless the certificate is valid for “domain.com” which is most cases it is not. Even if you had a wildcard certificate for “*.domain.com”, it would not prevent you from getting a warning because this specifies any subdomain is valid, but does not cover just the domain by itself. Your user will always get a warning message. You may think you could write a policy on your load balancer to do the redirect but that will not work. The reason is a TCP connection has to be established first and during the SSL handshake before the connection is fully established is when the user gets the warning. The policy will never run as a result until after the user accepts the warning message and the SSL handshake is completed.

There are a couple of ways to get around this. The cleanest way of doing it is using a Subject Alternate Name (SAN) SSL certificate. A SAN cert allows you to specify exactly what domains you would like the certificate to be valid for. The price on SAN certs is much higher than a standard certificate or even a wildcard certificate and they are only available from a handful of certificate authorities at this time. They require a lot more validation typically and the validation term is usually a bit lower than a standard certificate.

Let’s compare a SAN SSL certificate vs. a Wildcard SSL certificate. A single SAN cert would cover me for the following if I wanted it to:

domain.com
www.domain.com
www.myotherdomain.com

while a single Wilcard SSL cert would only be able to do the following:

*.domain.com

It is possible to combine both so you could get a SAN + Wildcard SSL certificate that will cover all of the above. To my knowledge, the only certificate authority that currently offers them is DigiCert. They have a pretty nice write up about their cert here:

http://www.digicert.com/ssl-support/wildcard-san-names.htm

I once asked a VeriSign managed PKI rep why they don’t offer wildcard + SAN like DigiCert and he went on for 5 minutes about ICAAN policy. Why one cert authority can issue while another won’t, I am guessing it likely has to do with acquisition costs for them. You have to remember, VeriSign is still trying to convince people to buy SGC SSL certs (for legacy browsers) at a premium price when in my opinion, SGC has been obsolete for years now and any web/network engineer that knows about SSL encryption has anything below 128 bit locked down anyhow. But I guess if people will buy it, they will sell it. Either there just hasn’t been enough demand or their costs are too high for wildcard + SAN certs. Not trying to pick on VeriSign, just making an observation. I still purchase certs from them regularly (non-SGC of course). Most every large organization does, but sometimes the wildcard + SAN might be the best solution for you. I know some CDN providers like Akamai sometimes have a need for them and purchase through DigiCert.

Getting back on subject, a second way of getting around this is purchasing two standard SSL certifcates and putting them on different IPs. One will do a 301 redirect to an another. So you would have to purchase an SSL cert for domain.com AND www.domain.com.

Let’s look at some real world examples to better understand this. I am going to compare Chase, Bank of America, Citrix, and Amazon.com to show the different types of solutions for this problem.

Chase.com
Chase uses SAN certs from VeriSign. Again, SAN certs are a very pricey approach but the easiest. Your configuration on your load balancer and servers is minimal since one SSL cert will take care of everything. This will result is less overhead on your load balancers as well which is a big plus when you serve up millions of connections.

Go to https://chase.com, it will immediately do a 301 redirect to https://www.chase.com. Here is a screenshot showing the 301 redirect to https://www.chase.com and the certificate which shows the 4 Subject Alt Names Chase’s cert is valid for:

As you can see, the certificate is valid for www.chase.com, chase.com, www.yourchasefreed.com, and yourchasefreedom.com. You will also notice that the cert is only valid for a 1 yr period. Typically CAs will try and limit you to a max of 2 years on SAN SSL certs and EV SSL certs but it is always best practice to keep them to 1 year just in case your cert keys were compromised. A SAN cert can have many domains leaving you very exposed, so it is a good idea to change it every year for this reason.

Bankofamerica.com
Bank of America uses standard SSL certs from VeriSign. Cheaper but a little extra configuration involved on the load balancer. Let’s take a look at what happens when you visit https://bankofamerica.com:

You will be 301 redirected to https://www.bankofamerica.com just as you were with Chase. But this time using a standard SSL cert with one CN name as you saw above. You can verify this by pinging bankofamerica.com which resolves to 171.159.100.173 and typing just the IP into the browser over SSL as https://171.159.100.173. This will give you an SSL error before the 301 redirect gets a chance to run and you can view the cert:

So just to be clear, typing https://bankofamerica.com will use the above cert which is valid, will complete the connection, run a policy that will do a 301 redirect to https://www.bankofamerica.com, cert for https://www.bankofamerica.com is validated, then the connection is completed and the page is served up. A perfectly good way around the problem using standard SSL certs as opposed to SAN SSL certs.

Citrix.com
Citrix is another example of using the standard SSL certificate redirect method much like Bank of America. But they do not leverage the 301 redirect at all. Instead they use a 302 temporary redirect pointed directly to their index page. The 302 is likely done for SEO purposes so the “www.citrix.com” domain is indexed in search results and not the “www.citrix.com/lang/English/home.asp” destination page. In this case, a Netscaler with either a responder policy or content switching policy will send you to https://www.citrix.com or https://citrix.com which both display the same content. They are on different VIPs with different SSL certs. One is a standard cert while the other a wildcard. They could have used two standard certs if they wanted to as well of course but in this case, they may use the wildcard for other subdomains, GSLB, etc. and the standard cert is meant to solely capture those users typing in an incorrect URL:

Amazon.com
Woops! The world’s largest online retailer does not seem to be redirecting customers that arrive on an irregular URL over SSL properly. If you go to https://amazon.com, it will give you a “This Connection is Untrusted” error. It’s not the end of the world but still, a user should never see a message like this.

In IE for example, it will say “There is a problem with this website’s security certificate”. Again, not the end of the world. But in Chrome, the error will have a bright red background and say “This is probably not the site you are looking for!
You attempted to reach amazon.com, but instead you actually reached a server identifying itself as www.amazon.com. This may be caused by a misconfiguration on the server or by something more serious. An attacker on your network could be trying to get you to visit a fake (and potentially harmful) version of amazon.com. You should not proceed.”

Yikes! Do you really want your users to see a message like this?

Interesting to note however, going to https://www.amazon.com does a 301 redirect to http://www.amazon.com. Obviously Amazon wants the customer to use HTTP rather than SSL on their homepage. Much less overheard on their load balancers plus they don’t have a login box directly on their home page so no need for SSL to begin with. This is thought out well but I am puzzled as to why they don’t do anything about https://amazon.com. As you saw with Chase, Bank of America, and Citrix, the different solutions are very easy to implement.

I hope this article has been helpful. Please post a reply if you have a different solution to the problem that I did not cover. I would love to hear your insights.

Citrix is now offering Netscaler VPX Express licenses with throughput increased from 1 Mbps to 5 Mbps! This will help tremendously in test lab environments for a lot of people. Most of my important test environments where I need some extra bandwidth were on VPX 10 or VPX 200 licenses already but now with a 5 Mbps throughput VPX Express for free, it gives me a little more wiggle room before I need to shell out the cash for these licenses.

That’s not all! Now those of us that are existing Netscaler customers are entitled to VPX Developer edition licenses with Platinum edition features! This makes testing features like integrated caching for example easier without having to pay for a license first. Sadly you are limited to 1 Mbps but it should be more than enough to test low bandwidth apps against Platinum features. You can always get a 90 eval though which has a 1 Gbps limited if you really need the extra bandwidth for testing.

Good move on Citrix’s part! This how how you corner the application delivery market and then stay on top. Give it away for free and let people see how powerful the Netscaler platform is first hand.

Read more about the changes on the Citrix Community blog here.

In a worst case scenario and all your web servers have failed, what do you do? You could have a standby group of servers or CDN on or off premise to pick up the load or at least display a maintenance page but this is worst case scenario. A catastrophic failure and ALL your servers are down due to a code issue, server configuration issue, database issue, virtual infrastructure failure, SAN failure, maintenance being performed on all servers at once (I hope not on purpose), virus outbreak, or whatever else kind of horrible scenario you can think of. You get traffic all the way up to the Netscaler appliance but since your vserver is down, the user’s browser will timeout as if your company fell off the face of the earth. This is very unprofessional for any organization. Users timing out or seeing a “page could not be displayed” error is unacceptable.

So the solution is to have the Netscaler display a maintenance page with the code hosted on itself somehow. I tried several different methods including content filtering and responder policies using HTML. Originally I even thought I could leverage integrated caching to serve up cached pages and static content like images. I settled only using a responder policy initially which worked. Citrix even has a very nice knowledge center article (CTX117337: How to Configure a Maintenance Web Page by using the Responder Feature of the NetScaler Appliance) which is located here:

http://support.citrix.com/article/CTX117337

In a nutshell, what the author of the article wrote is basically more or less the same conclusion I reached as well. I just did it via GUI and that is what I will show you below. But I was not happy with the result. Keep reading and you will see why. FYI, I did all the screenshots below on an NS 9.1 appliance but it is the same procedure on NS 9.2 or any other version.

1. I am going to assume you have servers, services/service groups, and a vserver already that is UP and running. I will call them the following in this example:

vserver – lb_vsver_mywebsite
service group – svcgrp_myservicegroup
server – svr_mywebserver

Excuse the redactions in my screenshots please, I had some other configurations on this test appliance and I don’t want to confuse you with it:

vserver:

service group:

server:

2. Now create a backup vserver for your existing live vserver. In this example, I have called it “lb_vsvr_bkup_mywebsite”. But instead of giving it an IP, just uncheck directly addressable. This will cause the IP area to become greyed out:

backup vserver:

When you click Create, it will show up as running on the IP 0.0.0.0 like below:

3. Now you need to create a service that is always UP and bind it to this backup vserver so that it will always remain UP. Just go under Load Balancing > Services, and click Add. Then create a service called “svc_maintpage” but for the Server, type in the localhost IP of 127.0.0.1, add a ping monitor, and press create.

4. Now go back to your backup vserver and bind this new service to it. Immediately after clicking OK, the backup vserver should go into an UP state. You might need to refresh your window if it doesn’t.

5. Now double click on your live vserver and under the Advanced tab, choose “lb_vsvr_bkup_mywebsite” for the Backup Virtual Server option and press OK:

6. Now under Responder > Action, click Add to create a new action. This is where you get to put some HTML and CSS. It must be very basic, all parenthesis have to be removed when using CSS in the HTML body or it will give you can error, and the whole policy must be under 255 characters total. I will name mine “action_mywebsite_maint_page” and here is an example of my policy I will use with it:

"HTTP/1.0 200 OK" +"\r\n\r\n" + "<html>
<style type=text/css>
<!--
.mywebsitefont {
  font-size: 24px;
}
-->
</style>
<body class=mywebsitefont>Sorry, our website is currently not available. 
Please try again later.</body></html>" + "\r\n"

7. Now under Responder > Policy, click Add to create a new policy that will call on the action you just created. In this example, all we need is for the HTTP request to be valid and we will display the maintenance page. I will name it “resp_policy_mywebsite_down” in this example. Choose the action you just made in the Action drop down and for the expression, just put:

HTTP.REQ.IS_VALID

8. Now go back to the Load Balancing folder and double click your backup vserver and bind the responder policy to it like below:

9. Now to test. Open up your website in a browser and it should display as normal right now. Now login to your webservers and turn off your websites. Immediately your live vserver should say DOWN for the State but the Effective State should remain UP. This is because all traffic is being forwarded to your backup vserver you specified earlier which is set to always be up:

Refresh your browser and you should now see the maintenance page you created like below:

As you can see, a simple HTML page like above is not very professional. We need more HTML/CSS than 255 characters to work with and we need images working to make it look professional. At least it is better than a page timeout though!

Now with a content filtering policy, you don’t have to worry about a character limit. You can get away with putting HTML/CSS in a content filter policy. But again, where do the images come from?

I decided to call Citrix and see if they have run into a request like this. They had not. Now off the bat both techs I spoke to said what I was trying to do is not supported by Citrix. A Netscaler is not designed to do this. But luckily the second tech Brian at Citrix Support was just as enthusiastic about getting something to work as I am and wasn’t going to give up easily so we went over a few scenarios. The Netscaler does have an Apache web server on board, that is how the admin GUI is display to you. It is also how the Access Gateway portal is displayed to the end user. We needed to figure out a way to leverage the Apache web server on board the Netscaler to host our images, HTML, CSS, etc. The initial thought was to overwrite the Access Gateway portal and create a responder policy that would do a redirect to an Access Gateway vserver you create. The negatives here are that you are limited to SSL traffic only, have to worry about having a valid cert, you can’t bind all the policies you might need to it like you can a load balanced VIP, etc. I didn’t feel that comfortable destroying functionality to gain other functionality either.

In the end, the solution was easy and did not require overwriting the Access Gateway portal. We can host our HTML, CSS, and images on the Netscaler itself and point Apache at it. Brian did a quick proof of concept in his lab. Then I improved on it a bit. Here is the end result which I am sure a lot of you will find pretty handy in your organizations. Steps 1 through 5 are the same as above. Then from there, begin these steps:

1. First we need to get our HTML, CSS, and images on the Netscaler. WinSCP into your Netscaler and go to “/netscaler/ns_gui”. The folders you see called admin_ui, vpn, etc. are what host the Netscaler Admin GUI and Access Gateway respectively. So you have the option of putting something in the root of this folder or even create a separate folder here if you want. In my case, I decided to put a “maintenance.htm” in the root and also create a folder called “static” that will host most static content like CSS and images.

2. Now under Responder > Action, click Add to create a new action. Very important, make sure to change the type from Response to Redirect. The action should be the following (with parenthesis included):

"http://www.mywebsite.com/maintenance.htm"

3. Now under Responder > Policy, click Add to create a new policy that will call on the action you just created.. Your responder policy will need to allow the maintenance page, plus CSS, .gifs, and .jpgs you might use. So the policy I will use is:

!HTTP.REQ.URL.CONTAINS("maintenance.htm") && !HTTP.REQ.URL.CONTAINS(".gif") && !HTTP.REQ.URL.CONTAINS(".jpg") && !HTTP.REQ.URL.CONTAINS(".css")

4. Now go back to the Load Balancing folder and double click your backup vserver and bind this new responder policy to it like I did below:

Now if you disable your service groups and check your maintenance page again, you can see how the website displays the full page with nice HTML, CSS, and images. In this example, I borrowed the Sears.com maintenance page. Notice how showing your company logo keeps your branding intact even on a maintenance page which is the correct way to handle a website issue. Tell your users you are aware of the problem and offer alternatives in the meantime (static links along the bottom to other servers that are up and offering content in this example). You don’t have to go that far but it’s always nice to let your user base know you haven’t disappeared and your infrastructure is solid. This is very professional and above all, automated!

The only problem here is that when your website is back up, users will still be refreshing on this maintenance.htm page. They will get a 404 error. So you have four options. I usually prefer number 4 personally but it all depends on your needs:

1. Change your maintenance.htm page to say index.htm or whatever page is the default page of the root of your website so when they refresh once the vserver is back up, they will get the live page. You will need to WinSCP into your Netscaler again and change the maintenance.htm file name as well as change it in your Responder Action. The issue here is if let’s say you are using .NET, you can’t call it index.aspx because Apache on the Netscaler can’t parse it.

2. Just create a link on the page that says “Click Here to Try Again” which is pointed at the correct index page. This assumes the end user will actually click the link instead of hitting refresh. You can’t be 100% sure they will do this.

3. Create a maintenance.htm page on your servers and then set IIS, Apache, or whatever web server you use to do a 301 redirect to your live index page. You can leverage the Netscaler to do the redirect too of course.

4. My preferred method. Create a new responder policy saying any maintenance.htm should automatically redirect to index.aspx and bind it only to your real vserver. That way anyone that requests that page when your servers are up will always be redirected to your index page. In this example, I will call my live site’s index page index.asp and call the action policy “action_mywebsite_index_redirect”. I will also make it redirect to SSL in this example because there is a login box on the index.asp page and I want to keep it secure using https:

I will call the responder policy “resp_policy_index_redirect” and for the expression, tell it to redirect any requests to “/maintenance.htm”:

HTTP.REQ.URL.CONTAINS("/maintenance.htm")

Now bind this to your live vserver:

Now you can test it by disabling and enabling your servers or service groups. It should transition automatically between your maintenance page and the live index page.

One thing I would like to point out. On any of your Responder Policies or Actions, you can always view the hit counter to see if the policy or action is being invoked. This might help you when you are setting this up initially and something goes wrong and you want to see if the policy or action is being hit:

So there it is. Your Netscaler is now an emergency web server that automatically puts up a professional looking maintenance page in a worst case scenario when every backend web server you have is down. A big thank you to Brian at Citrix for the help! If anyone can think of any improvements to this process or has any trouble with it, please reply I would love to hear about your experience.