Cheat Sheet

This is a quick reference guide/cheat sheet of links and commands every Citrix or VMware engineer should know about. I will remove stuff that gets deprecated so the page doesn’t get too cluttered.

Citrix living CTXs
These are critical parts of Citrix infrastructure and you always have to keep up with the latest news on these:

  1. http://support.citrix.com/article/CTX139331 – Citrix Virtual Desktop Handbook 7.x
  2. http://support.citrix.com/article/CTX127939 – XenDesktop 7.x SQL Database Sizing and Mirroring Practices
  3. http://support.citrix.com/article/CTX131239 – XenDesktop and PVS Hypervisor support
  4. http://support.citrix.com/article/CTX127030 & http://blogs.citrix.com/2013/09/22/citrix-consolidated-list-of-antivirus-exclusions/ – Citrix & Antivirus settings
  5. http://support.citrix.com/article/ctx129229 – Recommended Hotfixes for XenApp 6.x on Windows Server 2008 R2
  6. http://www.citrix.com/events/citrix-user-group.html – Upcoming Citrix User Group meeting in your city
  7. http://discussions.citrix.com/topic/357800-template-exchange-studio-templates-%E2%80%93-help-needed-out-of-the-box-configuration-sconmsg -d current -g pol_hitsfor-xendesktop-and-xenapp/ – XenDesktop/XenApp 7.6 Citrix Studio Template Exchange
  8. http://support.citrix.com/article/CTX127998 – SQL Database Access and Permissions Necessary for XenDesktop & XenApp 7.x
  9. https://www.citrix.com/go/citrix-developer/netscaler-developer-community/howto-guides.html – Citrix NetScaler Common Configuration How To guides
  10. http://support.citrix.com/article/CTX208792 – NetScaler Cheat sheets for XenMobile Integration
  11. http://support.citrix.com/article/CTX208788 – XenMobile WorxMail Cheat Sheet
  12. http://support.citrix.com/article/CTX208791 – XenMobile Device Enrollment Cheat Sheet
  13. http://support.citrix.com/article/CTX212665 – XenMobile Client Certificate Authentication Cheat Sheet
  14. https://support.citrix.com/user/alerts – Citrix product alerts. Select your products and get important updates as they happen.

 

VMware living KBs
These are critical parts of VMware vSphere infrastructure and you always have to keep up with the latest news on these:

  1. http://www.vmware.com/resources/compatibility/search.php – VMware Compatibility Guide (host server, guest OS, storage, etc.)
  2. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2100429 – Host operating system support information
  3. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009918 – Rebuilding indexes to improve the performance of SQL Server and Oracle vCenter Server databases
  4. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007453 – Reducing the size of the vCenter Server database when the rollup scripts take a long time to run

 

Microsoft living KBs

  1. http://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx – Microsoft Anti-Virus Exclusion List (very important for Citrix as well. Things like MMC, Powershell, DAT files, etc. should be excluded or apps like Citrix Studio that are heavily dependent on these will be extremely slow. Symantec Anti-virus (SAV) or Symantec Endpoint Protection (SEP) for example will rtvscan.exe or ccSvcHost.exe on each and every user .dat file when opening Citrix Studio.)

 

AppSense DesktopNow

  1. https://www.myappsense.com/Knowledgebase/TN-150728.aspx – Recommended AppSense DesktopNow anti-virus exclusions (login required)

 

Citrix XenDesktop and XenApp Powershell cmdlets:

  1. Enable Citrix powershell snap-ins for XenDesktop/XenApp 7.6:
    Add-PSSnapin Citrix.*.Admin.V*
  2. Enable Citrix powershell snap-ins for XenApp 6.5:
    Add-PSSnapIn citrix.xenapp.commands
    Add-PSSnapIn citrix.common.Commands
    Add-PSSnapIn citrix.common.groupPolicy
  3. To get a listing of all your XenApp 6.5 servers:
    get-xaserver | select servername > c:\temp\XenApp_VM_list.txt
  4. To get a listing of XenDesktop logged in users:
    get-brokersession | select UserName > c:\temp\logged_in_users.txt
  5. To get a listing of ALL XenDesktop/XenApp 7.6 VMs:
    get-brokermachine -maxrecordcount 2000 | select machinename > c:\temp\VM_list.txt
  6. To send a reboot notification message to a specific desktop group on XenDesktop/XenApp 7.6:
    Add-PSSnapin citrix*
    $sessions = Get-BrokerSession -MaxRecordCount 5000 | Where {$_.desktopgroupname -match "Virtual Desktop Group 1"}
    Send-BrokerSessionMessage $sessions -MessageStyle "Information" -Title "REMINDER: Virtual Desktop Reboot @ 11 PM" -Text "REMINDER: Virtual Desktops will be rebooted Saturdays @ 11 PM. Please save your work and logoff prior to this when you go home for the day. If you work from home you will be logged off at 11 PM but can immediately log back in after the 11 PM reboot."
  7. To get a list of all published applications being used along with associated user names in a XenApp 7.x environment:
    Add-PSSnapin Citrix.*.Admin.V*
    get-brokersession -maxrecordcount 20000 | select ApplicationsInUse, DesktopGroupName, UserName, UserUPN, ClientName, ConnectedViaIP, AppState, EstablishmentTime | Sort-Object ApplicationsInUse > c:\temp\citrix-apps-launched-right-now.txt

 

VMware PowerCLI scripts:

  1. Reboot specified VMs in a controlled fashion. I use a modified version of James Green’s PowerCLI script. Comes in very handy for Citrix PVS/Atlantis USX environments that require weekly reboots. It is always better to initiate reboots at the host level in case any VMs are stuck on boot and won’t respond to agent or OS based reboot instructions. I’ll link you directly to James’s work instead of posting my customized version for PVS, it’s essentially the same:
    Scheduled VM Reboots with PowerCLI

 

Citrix NetScaler commands:

  1. Generate an SSL key and csr from a conf file for an SSL SAN certificate (with expectation you have created a valid .conf file with all SAN names and uploaded already). Your conf file should look something like this:

    After entering shell:
    cd /nsconfig/ssl
    followed by:
    openssl req -new -newkey rsa:2048 -keyout 2015_KEY_yourdomain.key -out 2015_CSR_yourdomain.csr -config 2015_REQ_yourSANcert.conf
    and enter a PEM passphrase for the key file. Send the resulting CSR file in to your SSL cert vendor. They will send back a .cer file. Go to step 2 below to convert to a pfx with the key included. Go to step 3 to convert pfx to pem so you can install it on the NetScaler and actually use the cert.

  2. Convert a .cer SSL cert and SSL key to .pfx format using openssl:
    openssl pkcs12 -export -out 2015_CER_yourdomain.pfx -inkey 2015_KEY_yourdomain.key -in 2015_CER_yourdomain.cer
  3. Convert a .pfx (SSL cert and SSL key) to .pem format using openssl:
    openssl pkcs12 -in 2015_CER_yourdomain.pfx -out 2015_CER_yourdomain.pem
  4. If you get an “Invalid private key, or PEM pass phrase required for this private key” error message when attempting to install your .pem and private key on your NetScaler, this is because of a hidden space issue with the key. You will not be able to bind the cert to your vserver, it will complain it is not a Server Certificate when you attempt to bind. Run the commands below in your NetScaler SSH window to convert your key and re-install the cert using this new key. It will install without error this time. Also click the little black arrow next to the cert after installing which will expand it. You should see your new key in the “Key File Name” section which confirms the cert is linked to the key properly:
    shell
    cd /nsconfig/ssl
    openssl rsa -in 2015_KEY_yourdomain.key -out 2015_KEY_yourdomain_2.key
  5. Watch NetScaler Gateway logins in realtime (AAA debugging) and log to a file:
    shell
    followed by:
    cat /tmp/aaad.debug | tee /tmp/aaa-log-mm-dd-yyyy.log
    **TIP: I absolutely LOVE to use CMTrace to run through these logs quickly. Will highlight rejects and errors automatically. I highly recommend this for any NetScaler engineer.
  6. See all connections to the NetScaler:
    shconnectiontable
  7. See all Established connections to the NetScaler from a certain IP and omit Monitors:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR"
  8. See all connections to the NetScaler from a certain IP and only display SSL:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established && svctype == SSL"
  9. See all connections to the NetScaler that are NOT Monitors or SSL:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR && svctype != SSL"
  10. Watch NetScaler policy hits in realtime:
    shell
    followed by:
    nsconmsg -d current -g pol_hits
  11. Watch latest ns.log events in realtime (even GUI clicks since they are “show” commands):
    shell
    followed by:
    tail -f /var/log/ns.log
  12. Change the NetScaler theme from a Custom theme to Default theme via CLI if after a firmware upgrade you are unable to login via GUI anymore (“Login command failed over API. Reason: Response not of type test/xml: test/html” bug). Once you’re back in you can change back to custom using the GUI. Make sure your password does not contain special characters as that can prevent login too (“/login/do_login” bug):
    set vpn parameter UITHEME DEFAULT
    save ns conf
    reboot
  13. Change the password on nsroot or other user account via command line:
    set system user nsroot MyNewPassword54321
    save ns conf
  14. NetScaler licensing is via Host ID, Serial Number, or MAC Address depending on the platform. Sometimes the System page won’t show the MAC address under the Host ID. To find the true MAC address (FlexNet host ID) you need to allocate licenses to, go to:
    shell
    followed by:
    lmutil lmhostid
  15. Find and delete ghost bindings in the config that don’t show in GUI (AppFlow used as example):
    sh run | grep -i af_policy_vsrv_gw_192.168.1.10
    and the bound vserver will show. To unbind the policy:
    unbind vpn vserver vsrv_gw -policy af_policy_vsrv_gw_192.168.1.10
    To find an object (AppFlow collector used as example):
    sh run | grep af_collector_192.168.1.20
    To remove the object:
    rm appflow af_collector_192.168.1.20
  16. Troubleshooting AppFlow from NetScaler to NMAS or Insight Center. The most common ports NMAS requires to communicate with your NetScaler are:

    TCP 22,80,443
    UDP 161,162, 514, 4739

    TCP ports are used for discovery and configuration changes. The UDP ports are for SNMP (161,162), SYSLOG (514), and AppFlow (4739). AppFlow is an IPFIX based protocol where traffic is multicast and will flow from the NS to the NMAS appliance one way.

    In this example:
    192.168.150.20 = DMZ NetScaler
    172.12.40.10 = NMAS appliance in another subnet seperated by firewall

    On your NetScaler run:
    shell
    nstcpdump.sh -nn host 172.12.40.10 and port 4739

    If you see traffic being sent to the NMAS appliance and no response, this is good and how it should work.

    On your NMAS appliance run:
    shell
    tcpdump host 192.168.150.20 and port 4739

    If you do not see any traffic being received, your firewall is likely blocking it and need to investigate further.

  17. Check to see what Cavium Nitrox chips (SSL accelerator cards) your physical NetScaler, such as an MPX NetScaler has (look for something like “CVM N3” in the output meaning Nitrox III processors):
    sh hardware

    If you see something like “CVM 1620” those are the older generation Nitrox PX cards (N2 cards).

    If it’s an SDX appliance, you need to SSH into the XenServer, not the SVM. You must login with root, not nsroot. Then run:
    lspci -vv | grep -i cavium

 

Splunk syslog search filters for NetScaler:

  1. Find successful NetScaler Gateway logins with device/browser, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" SSLVPN LOGIN NOT (HTTPREQUEST)
  2. Find failed NetScaler Gateway logins with device/browser, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" AAA LOGIN
  3. Create a report of Top 20 SSL ciphers being used on a NetScaler Gateway vserver:
    source="YourNetScalerSource" index="YourNetScalerIndex" CipherSuite VserverServiceIP xxx.xxx.xxx.xxx| top limit=20 SSLCipher
    where SSLCipher is the following custom field extraction:
  4. Find NetScaler Gateway ICA launches with the name of the app/desktop, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" SSLVPN ICASTART

 

Atlantis ILIO/USX commands
These Linux commands are useful but depend on your storage setup and if you are using ILIO Diskless or ILIO Persistent. These are commands that I personally use so modify as needed.

  1. ILIO appliance snapclone info assuming disk 1 is snapclone:
    dstat -D sdb -cdln --disk-util
  2. ILIO appliance snapclone partition info:
    dstat -D sdb2 -dsr --disk-util
  3. ILIO VM total disk usage:
    df -h /exports/ILIO_VirtualDesktops
  4. ILIO appliance VM info (network, disk, resource with resource limit on nfs stack on dom0):
    dstat -D dm-0 -ndr --nfsd3
  5. ILIO appliance VM info (time, load, network, cpu, disk, resource):
    dstat -D dm-0 -tlncdr --nfsd3
  6. ILIO appliance OS info assuming it is disk 0 (network, disk, resource on disk):
    dstat -D sda -ndr --disk-util
  7. ILIO appliance VM snapclone info assuming it is disk 1 (network, disk, resource on disk):
    dstat -D sdb -ndr --disk-util

 

PowerShell Oneliners
Just open a PowerShell window and drop each line in to get a result.

This first set of one liners returns common redirected folders. If you use AppSense for example, you will likely be redirecting these folders to a file share for each user. You can use these one liners as a base for advanced scripts on reporting redirected user folders for all your users:

  1. Display the redirected Desktop folder:
    [Environment]::GetFolderPath("Desktop")
  2. Display the redirected My Documents folder:
    [Environment]::GetFolderPath("MyDocuments")
  3. Display the redirected My Music folder:
    [Environment]::GetFolderPath("MyMusic")
  4. Display the redirected My Pictures folder:
    [Environment]::GetFolderPath("MyPictures")
  5. Display the redirected My Videos folder:
    [Environment]::GetFolderPath("MyVideos")

 

AutoHotkey Scripts

  1. Left click mouse every 2 seconds in a loop. Good foundation to do more advanced stuff with.

 

1 Comment

  1. Carlos Valdes

    December 31, 2016 at 5:14 AM

    Thanks Jason, this is really good. Looking forward to work with you.

Leave a Reply

Your email address will not be published. Required fields are marked *