JasonSamuel.com

@Scott

Hi Scott,

Here is what I do when I update Netscaler firmware in an HA cluster. This is all from memory so please let me know if something needs clarification:

1. First thing is read the release notes and see what all changes are happening with the update. You always want to be aware of what kind of impact an update will have on features you might be dependent on. You will also know what to test after the update and verify if it’s all still working or not.

2. Test the firmware update on your Test environment if you have one. VPX Express edition is a FREE download from Citrix and is perfect for testing on. Set them up in an HA pair and practice if this is your first time updating an HA environment. They work on both XenServer and VMware ESX/vSphere just fine.

3. Schedule a maintenance window for your Production environment. Firmware updates have been pretty seamless for me but I always like to make sure I have a scheduled maintenance window incase anything goes wrong. The less traffic through the Netscaler, the better.

4. Go to your secondary Netscaler (passive node that is not handling traffic). Go to System > High Availability > then click on your Secondary node to bring up the Properties dialog box. Click “STAY SECONDARY (Remain in Listen mode)”. I also like to uncheck HA Synchronization and HA Propagation. Then press OK and hit Save. What this does is prevent your Secondary node you are going to be working on from becoming primary by accident during the firmware update.

5. Go to your primary Netscaler (active node that is handling traffic). Go to System > High Availability > and select “STAY PRIMARY”. This is just a precaution I always do. The likelihood of working on the passive node causing a failover from the active node to the passive node is pretty slim but I always like to be careful.

6. Now that your HA failover and synchronization is effectively “paused”, we can begin the update on the passive node. I just want to point out to you that in an HA pair, if you have 2 different firmware versions running, it will detect an HA version mismatch and the node with the most current firmware will automatically become secondary (listen mode). So we always want to make sure and update the secondary node first before moving onto the active node with live traffic on it. Just as a precaution, open command prompt and start a continuous ping on one of your VIP IPs and leave it minimized. This way you will know if you have an outage. You shouldn’t, but just do it to be cautious and keep an eye on things during the update process.

7. Now begin your firmware update on the passive node as described in my article. Make sure to verify the firmware version in the top right corner reflects the update and the system is still secondary after the reboot. Once you have verified everything, turn HA, synch, and propagation back on.

8. Now go back to your primary node and enable HA that you had disabled in step 5 above.

9. Now you have an option. The first option is that you can update the primary node next. Most people do this step next. I prefer to do it later once I have passed some traffic through the newly updated Netscaler and verify everything is good. This is because I want to have the option of failing back to the old firmware immediately if I have an issue. If you want to go with option 1, just skip to step 11 below. If you want to go the route I take and test it first, move onto step 10 below.

10. Do a forced failover making your newly updated Netscaler the primary node. Traffic will immediately begin passing through it. Watch your traffic and do testing! Verify all your services are up, all your VIPs are up, all your apps are responding normally, your traffic looks good, etc. This is the only chance you have to failback to the node with the old firmware so it is critical you test everything. I have gone as long as a week running new firmware on a primary node without updating the passive node just so I have a safety net and give application owners time to test their apps.

11. When you are finally ready to update the primary node, perform the same steps as above you did on the passive node. Yes you can do the update while there is traffic on your node. Nothing will happen to the traffic until you tell it to reboot. When you reboot, it will automatically failover to the passive node (just make sure you don’t disable HA like you did above earlier). There should be no outage (verify this in the command prompt where you are still pinging your VIP). Once the node is back up, verify everything looks good and do a manual failback. Now both your nodes should be updated and the node that was originally primary when you began the update process is back to being primary again.

That’s it! Let me know if you have any questions. I urge you to setup 2 VPX Express appliances in an HA pair and test this before doing it on your production Netscalers just to make sure you have the process down and don’t have any surprises. Also don’t forget to take a backup of your ns.conf before beginning any firmware updates just incase. Let me know how it goes for you.

Jason