Cheat Sheet

This is a quick reference guide/cheat sheet of links and commands every EUC (End User Computing) or security focused engineer should know about. I will remove stuff that gets deprecated so the page doesn’t get too cluttered.

Citrix living CTXs

These are critical parts of Citrix infrastructure and you always have to keep up with the latest news on these:

  1. http://support.citrix.com/article/CTX139331 – Citrix Virtual Desktop Handbook 7.x
  2. http://support.citrix.com/article/CTX127939 – XenDesktop 7.x SQL Database Sizing and Mirroring Practices
  3. http://support.citrix.com/article/CTX131239 – XenDesktop and PVS Hypervisor support
  4. http://support.citrix.com/article/CTX127030 & http://blogs.citrix.com/2013/09/22/citrix-consolidated-list-of-antivirus-exclusions/ – Citrix & Antivirus settings
  5. http://support.citrix.com/article/ctx129229 – Recommended Hotfixes for XenApp 6.x on Windows Server 2008 R2
  6. http://www.citrix.com/events/citrix-user-group.html – Upcoming Citrix User Group meeting in your city
  7. http://discussions.citrix.com/topic/357800-template-exchange-studio-templates-%E2%80%93-help-needed-out-of-the-box-configuration-sconmsg -d current -g pol_hitsfor-xendesktop-and-xenapp/ – XenDesktop/XenApp 7.6 Citrix Studio Template Exchange
  8. http://support.citrix.com/article/CTX127998 – SQL Database Access and Permissions Necessary for XenDesktop & XenApp 7.x
  9. https://www.citrix.com/go/citrix-developer/netscaler-developer-community/howto-guides.html – Citrix NetScaler Common Configuration How To guides
  10. http://support.citrix.com/article/CTX208792 – NetScaler Cheat sheets for XenMobile Integration
  11. http://support.citrix.com/article/CTX208788 – XenMobile WorxMail Cheat Sheet
  12. http://support.citrix.com/article/CTX208791 – XenMobile Device Enrollment Cheat Sheet
  13. http://support.citrix.com/article/CTX212665 – XenMobile Client Certificate Authentication Cheat Sheet
  14. https://support.citrix.com/user/alerts – Citrix product alerts. Select your products and get important updates as they happen.

 

VMware living KBs

These are critical parts of VMware vSphere infrastructure and you always have to keep up with the latest news on these:

  1. http://www.vmware.com/resources/compatibility/search.php – VMware Compatibility Guide (host server, guest OS, storage, etc.)
  2. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2100429 – Host operating system support information
  3. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009918 – Rebuilding indexes to improve the performance of SQL Server and Oracle vCenter Server databases
  4. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007453 – Reducing the size of the vCenter Server database when the rollup scripts take a long time to run

 

Microsoft living KBs

  1. http://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx – Microsoft Anti-Virus Exclusion List (very important for Citrix as well. Things like MMC, Powershell, DAT files, etc. should be excluded or apps like Citrix Studio that are heavily dependent on these will be extremely slow. Symantec Anti-virus (SAV) or Symantec Endpoint Protection (SEP) for example will rtvscan.exe or ccSvcHost.exe on each and every user .dat file when opening Citrix Studio.)

 

AppSense DesktopNow

  1. https://www.myappsense.com/Knowledgebase/TN-150728.aspx – Recommended AppSense DesktopNow anti-virus exclusions (login required)

 

Citrix XenDesktop and XenApp Powershell cmdlets

  1. Enable Citrix powershell snap-ins for XenDesktop/XenApp 7.6:
    Add-PSSnapin Citrix.*.Admin.V*
  2. Enable Citrix powershell snap-ins for XenApp 6.5:
    Add-PSSnapIn citrix.xenapp.commands
    Add-PSSnapIn citrix.common.Commands
    Add-PSSnapIn citrix.common.groupPolicy
  3. To get a listing of all your XenApp 6.5 servers:
    get-xaserver | select servername > c:\temp\XenApp_VM_list.txt
  4. To get a listing of XenDesktop logged in users:
    get-brokersession | select UserName > c:\temp\logged_in_users.txt
  5. To get a listing of ALL XenDesktop/XenApp 7.6 VMs:
    get-brokermachine -maxrecordcount 2000 | select machinename > c:\temp\VM_list.txt
  6. To send a reboot notification message to a specific desktop group on XenDesktop/XenApp 7.6:
    Add-PSSnapin citrix*
    $sessions = Get-BrokerSession -MaxRecordCount 5000 | Where {$_.desktopgroupname -match "Virtual Desktop Group 1"}
    Send-BrokerSessionMessage $sessions -MessageStyle "Information" -Title "REMINDER: Virtual Desktop Reboot @ 11 PM" -Text "REMINDER: Virtual Desktops will be rebooted Saturdays @ 11 PM. Please save your work and logoff prior to this when you go home for the day. If you work from home you will be logged off at 11 PM but can immediately log back in after the 11 PM reboot."
  7. To get a list of all published applications being used along with associated user names in a XenApp 7.x environment:
    Add-PSSnapin Citrix.*.Admin.V*
    get-brokersession -maxrecordcount 20000 | select ApplicationsInUse, DesktopGroupName, UserName, UserUPN, ClientName, ConnectedViaIP, AppState, EstablishmentTime | Sort-Object ApplicationsInUse > c:\temp\citrix-apps-launched-right-now.txt

 

VMware PowerCLI scripts

  1. Reboot specified VMs in a controlled fashion. I use a modified version of James Green’s PowerCLI script. Comes in very handy for Citrix PVS/Atlantis USX environments that require weekly reboots. It is always better to initiate reboots at the host level in case any VMs are stuck on boot and won’t respond to agent or OS based reboot instructions. I’ll link you directly to James’s work instead of posting my customized version for PVS, it’s essentially the same:
    Scheduled VM Reboots with PowerCLI

 

Citrix NetScaler commands

  1. Generate an SSL key and csr from a conf file for an SSL SAN certificate (with expectation you have created a valid .conf file with all SAN names and uploaded already). Your conf file should look something like this:

    After entering shell:
    cd /nsconfig/ssl
    followed by:
    openssl req -new -newkey rsa:2048 -keyout 2015_KEY_yourdomain.key -out 2015_CSR_yourdomain.csr -config 2015_REQ_yourSANcert.conf
    and enter a PEM passphrase for the key file. Send the resulting CSR file in to your SSL cert vendor. They will send back a .cer file. Go to step 2 below to convert to a pfx with the key included. Go to step 3 to convert pfx to pem so you can install it on the NetScaler and actually use the cert.

  2. Convert a .cer SSL cert and SSL key to .pfx format using openssl:
    openssl pkcs12 -export -out 2015_CER_yourdomain.pfx -inkey 2015_KEY_yourdomain.key -in 2015_CER_yourdomain.cer
  3. Convert a .pfx (SSL cert and SSL key) to .pem format using openssl:
    openssl pkcs12 -in 2015_CER_yourdomain.pfx -out 2015_CER_yourdomain.pem
  4. If you get an “Invalid private key, or PEM pass phrase required for this private key” error message when attempting to install your .pem and private key on your NetScaler, this is because of a hidden space issue with the key. You will not be able to bind the cert to your vserver, it will complain it is not a Server Certificate when you attempt to bind. Run the commands below in your NetScaler SSH window to convert your key and re-install the cert using this new key. It will install without error this time. Also click the little black arrow next to the cert after installing which will expand it. You should see your new key in the “Key File Name” section which confirms the cert is linked to the key properly:
    shell
    cd /nsconfig/ssl
    openssl rsa -in 2015_KEY_yourdomain.key -out 2015_KEY_yourdomain_2.key
  5. Watch NetScaler Gateway logins in realtime (AAA debugging) and log to a file:
    shell
    followed by:
    cat /tmp/aaad.debug | tee /tmp/aaa-log-mm-dd-yyyy.log
    **TIP: I absolutely LOVE to use CMTrace to run through these logs quickly. Will highlight rejects and errors automatically. I highly recommend this for any NetScaler engineer.
  6. See all connections to the NetScaler:
    shconnectiontable
  7. See all Established connections to the NetScaler from a certain IP and omit Monitors:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR"
  8. See all connections to the NetScaler from a certain IP and only display SSL:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && state == established && svctype == SSL"
  9. See all connections to the NetScaler that are NOT Monitors or SSL:
    sh connectiontable "ip == xxx.xxx.xxx.xxx && svctype != MONITOR && svctype != SSL"
  10. Watch NetScaler policy hits in realtime:
    shell
    followed by:
    nsconmsg -d current -g pol_hits
  11. Watch latest ns.log events in realtime (even GUI clicks since they are “show” commands):
    shell
    followed by:
    tail -f /var/log/ns.log
  12. Change the NetScaler theme from a Custom theme to Default theme via CLI if after a firmware upgrade you are unable to login via GUI anymore (“Login command failed over API. Reason: Response not of type test/xml: test/html” bug). Once you’re back in you can change back to custom using the GUI. Make sure your password does not contain special characters as that can prevent login too (“/login/do_login” bug):
    set vpn parameter UITHEME DEFAULT
    save ns conf
    reboot
  13. Change the password on nsroot or other user account via command line:
    set system user nsroot MyNewPassword54321
    save ns conf
  14. NetScaler licensing is via Host ID, Serial Number, or MAC Address depending on the platform. Sometimes the System page won’t show the MAC address under the Host ID. To find the true MAC address (FlexNet host ID) you need to allocate licenses to, go to:
    shell
    followed by:
    lmutil lmhostid
  15. Find and delete ghost bindings in the config that don’t show in GUI (AppFlow used as example):
    sh run | grep -i af_policy_vsrv_gw_192.168.1.10
    and the bound vserver will show. To unbind the policy:
    unbind vpn vserver vsrv_gw -policy af_policy_vsrv_gw_192.168.1.10
    To find an object (AppFlow collector used as example):
    sh run | grep af_collector_192.168.1.20
    To remove the object:
    rm appflow af_collector_192.168.1.20
  16. Troubleshooting AppFlow from NetScaler to NMAS or Insight Center. The most common ports NMAS requires to communicate with your NetScaler are:

    TCP 22,80,443
    UDP 161,162, 514, 4739

    TCP ports are used for discovery and configuration changes. The UDP ports are for SNMP (161,162), SYSLOG (514), and AppFlow (4739). AppFlow is an IPFIX based protocol where traffic is multicast and will flow from the NS to the NMAS appliance one way.

    In this example:
    192.168.150.20 = DMZ NetScaler
    172.12.40.10 = NMAS appliance in another subnet seperated by firewall

    On your NetScaler run:
    shell
    nstcpdump.sh -nn host 172.12.40.10 and port 4739

    If you see traffic being sent to the NMAS appliance and no response, this is good and how it should work.

    On your NMAS appliance run:
    shell
    tcpdump host 192.168.150.20 and port 4739

    If you do not see any traffic being received, your firewall is likely blocking it and need to investigate further.

  17. Check to see what Cavium Nitrox chips (SSL accelerator cards) your physical NetScaler, such as an MPX NetScaler has (look for something like “CVM N3” in the output meaning Nitrox III processors):
    sh hardware

    If you see something like “CVM 1620” those are the older generation Nitrox PX cards (N2 cards).

    If it’s an SDX appliance, you need to SSH into the XenServer, not the SVM. You must login with root, not nsroot. Then run:
    lspci -vv | grep -i cavium

 

Quick packet capturing/tracing commands on the NetScaler

Nstrace is a NetScaler script that will help you do a packet capture and is the gold standard for troubleshooting network traffic on a NetScaler. You can do it from the GUI or from a command line. If command line do not enter the “shell”. Always set packet size to 0 regardless of method chosen as this will capture each and every single packet regardless of what size the packet is. The “nstrace1.cap” or “nstrace1.pcap” files will be dropped into the “/var/nstrace” folder with a date and timestamp by default. Since much traffic these days is using SSL/TLS, capture the SSL keys using the “-capsslkeys ENABLED” switch which will save the keys to a separate file from the capture called “nstrace.sslkeys”. Or you can use the “sslplain” switch which will decrypt the trace on the fly so you don’t have to mess with keys at all.

  1. To begin capturing traffic:
    start nstrace -size 0 -mode sslplain
  2. To stop capturing traffic
    stop nstrace
  3. To show the status of the packet capture:
    show nstrace
  4. To create capture filters on a specific vserver you are troubleshooting:
    start nstrace -size 0 -mode sslplain -filter "vsvrname == lbvsrv_StoreFront"
  5. To capture all traffic to a destination IP:
    start nstrace -size 0 -mode sslplain -filter "DESTIP == 192.168.1.10"
  6. To capture all traffic hitting a source IP:
    start nstrace -size 0 -mode sslplain -filter "SOURCEIP == 192.168.1.10"
  7. To capture all traffic on a destination port:
    start nstrace -size 0 -mode sslplain -filter "DESTPORT == 443"
  8. To capture traffic from source IP on a specific port:
    start nstrace -size 0 -mode sslplain -filter "SOURCEIP == 192.168.1.10 && DESTPORT == 443"
  9. To do a cyclical nstrace capture to help troubleshoot intermittent issues and you don’t want to sit there all day babysitting a packet capture. Example, a new trace is captured every 60 seconds between the destination IP and source IP specified. It will create 50 of these files and then start overwriting them from oldest to newest. This will conserve drive space on the NetScaler while still allowing you to capture some traffic history. Just make sure to stop the nstrace as soon as the issue occurs so the files don’t get ovewrited:
    start nstrace -size 0 -mode sslplain -filter "CONNECTION.DSTIP.EQ(192.168.1.10) || CONNECTION.SRCIP.EQ(10.0.0.10)" -size 0 -nf 50 -time 60
  10. Capture traffic on a specific network interface, example interface 0/1:
    start nstrace -size 0 -mode sslplain -tcpdump ENABLED -filter CONNECTION.INTF.EQ("0/1")
  11. Capture traffic on a specific VLAN, example VLAN 3:
    start nstrace -size 0 -mode sslplain -tcpdump ENABLED -filter "CONNECTION.VLANID.EQ(3)"

Quick TCP dump commands on the NetScaler

This is another NetScaler script. It is not as good as doing an nstrace but can grab quick TCP dumps for initial troubleshooting. It will output directly to the Putty/SSH window so no need to view traces in Wireshark. You must enter “shell” first before running these commands. Press Ctrl+C to stop the dump after it has started. The “>” sign denotes the direction of traffic (source to destination). You cannot run nstrace and nstcpdump at the same time.

For troubleshooting SSL ICA proxy, if you try and dump traffic to/from your NetScaler Gateway vserver IP (or Content Switching vserver IP if using Unified Gateway) and you have MAC based forwarding disabled (the default setting), then traffic will often be showing going back and forth to your default gateway IP. If you enable MAC based forwarding, then traffic will be shown between the NetScaler and the client IP. MAC based forwarding sometimes “hides” other fundamental networking issues in an environment so I don’t like using it.

For XenDesktop/XenApp troubleshooting, if you try and dump traffic on the VDA IP then you’ll see the 2 way communication between VDA and NetScaler SNIP. This is something I find very useful for troubleshooting purposes.

  1. To show all traffic going in or out of the NetScaler:
    nstcpdump.sh
  2. To show MAC addresses plus the IPs (good to find what network devices are talking to NetScaler):
    nstcpdump.sh -ne
  3. To show all traffic coming to or going out of a specific IP (both source or destination):
    nstcpdump.sh host 192.168.1.10
  4. To show traffic sent to a destination IP:
    nstcpdump.sh dst host 192.168.1.10
  5. To show traffic from a source IP:
    nstcpdump.sh src host 192.168.1.10
  6. To show traffic conversation between 2 IPs (source and destination):
    nstcpdump.sh host 192.168.1.10 and host 10.0.0.10
  7. To dump this output to a capture file:
    nstcpdump.sh dst host 192.168.1.10 -w /tmp/tcpdump_output.cap
  8. To show the first 30 packets only and then stop:
    nstcpdump.sh -c 30 dst host 192.168.1.10
  9. To show all traffic on a specific port:
    nstcpdump.sh port 443
  10. To show all traffic to a destination IP on a specific port:
    nstcpdump.sh dst host 192.168.1.10 and port 443
  11. To show all traffic from a source IP on a specific port:
    nstcpdump.sh src host 192.168.1.10 and port 443
  12. To show all ICMP ping requests to a destination host (the ping-default monitor is a very common monitor used):
    nstcpdump.sh icmp and dst host 192.168.1.10

Quick ways to parse events on the NetScaler

Nsconmsg is an executable that will show you console messages on the NetScaler. You must be in “shell”. If you use it against the newslog you can grab historical events. DO NOT USE lowercase “k” as it is used in conjunction with another switch to overwrite what you are looking at. Only use uppercase “K” for reading.

• Ns.log is the system syslog file. It and related historical logs (ns.log.xx.gz) are in /var/log.

• Newns.log is the main log file. It and related historical logs (newnslog.xx.gz) are in /var/nslog. There will also be an ns.log here.

• Auth.log is the authentication and authorization log (AAA log) and is located in /var/log. This is different from the aaad.debug log you usually dump into /tmp.

If you ever grab the newslog .gz files and decompress them onto a Windows box they will have the extension newnslog.ppe.0, newnslog.ppe.1, etc. These are created by the packet engine (nsppe process). Use nsconmsg on the NetScaler to view these files.

  1. One of my favorites is to tail the ns.log so I can catch realtime events (tail shows the tail end of the log). It also shows GUI commands so it’s a great way to see what command line is executed on the NetScaler with each click:
    tail -f /var/log/ns.log
  2. You can also cat the ns.log to look at historical events:
    cat /var/log/ns.log
  3. Will show you what major events happened historically to vservers (up, down, probe/monitor failures, etc):
    nsconmsg -K /var/nslog/newnslog -d event
  4. You can look at older logs by doing. This will decompress the gz file and parse through the log for you:
    nsconmsg -K /var/nslog/newnslog.24.tar.gz -d event
  5. And further commands can be run directly against the decompressed newnslog like this:
    nsconmsg -K /var/nslog/newnslog.24 -d event
  6. Watch policy hits in realtime:
    nsconmsg -d current -g pol_hits
  7. Watch for SAML authentication issues (normal login will show saml_assertion_verify_success, saml_tot_dht_put_success, saml_tot_dht_get_notfound):
    nsconmsg -d current -g saml
  8. Look at historical SAML authentication:
    nsconmsg -d stats -g saml
  9. To look for IP conflicts (everything will be green and seem like it’s working on the appliance so the log is the only way to tell of IP conflicts from other systems on the network):
    nsconmsg -K newnslog -d | grep conflict

    And you will get entries like this which prove an IP conflict:
    IP address conflict 192.168.101.106! sent from ethernet address: 00:0a:49:c6:f6:40

    Take the conflicting MAC address above and go to the Coffer website at http://www.coffer.com/mac_find/ to verify which device the MAC belongs. In this example the MAC addresses that starts with prefix 00:0A:49 belongs to an F5 appliance. This means the F5 appliance has the same IP on it as one of the NetScaler vservers has as a virtual IP in most cases from my past experience troubleshooting these types of issues.

Quick Wireshark display filters to troubleshoot NetScaler

Note, these are display filters to use after you have captured a trace. These are not capture filters that you use while actively capturing traffic.

  1. To show all traffic to or from a source IP or destination IP:
    ip.addr == 192.168.1.10
  2. To show all conversation traffic between 2 IP addresses (source IP and destination IP):
    ip.addr==192.168.1.10 && ip.addr==10.0.0.10
  3. To show all http and dns traffic only (filtering by protocol):
    http or dns
  4. To hide all the ARP, ICMP, DNS noise that is common in packet captures:
    !(arp or icmp or dns)
  5. To filter traffic by port:
    tcp.port==443
  6. To display TCP resets:
    tcp.flags.reset==1
  7. To display all HTTP GET requests:
    http.request
  8. To display all HTTP GET requests and the responses:
    http.request or http.response
  9. To display all TCP packets that contain a certain word in the payload. In this example the User ID “jdoe”:
    tcp contains jdoe
  10. To display TCP retransmissions (good indicator of packet loss or slow performance):
    tcp.analysis.retransmission
  11. To display a specific User Agent, example “CitrixReceiver”:
    http.user_agent contains "CitrixReceiver"

Splunk syslog search filters for NetScaler

  1. Find successful NetScaler Gateway logins with device/browser, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" SSLVPN LOGIN NOT (HTTPREQUEST)
  2. Find failed NetScaler Gateway logins with device/browser, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" AAA LOGIN
  3. Create a report of Top 20 SSL ciphers being used on a NetScaler Gateway vserver:
    source="YourNetScalerSource" index="YourNetScalerIndex" CipherSuite VserverServiceIP xxx.xxx.xxx.xxx| top limit=20 SSLCipher
    where SSLCipher is the following custom field extraction:
  4. Find NetScaler Gateway ICA launches with the name of the app/desktop, user ID, and IP address:
    source="YourNetScalerSource" index="YourNetScalerIndex" SSLVPN ICASTART

 

Atlantis ILIO/USX commands

These Linux commands are useful but depend on your storage setup and if you are using ILIO Diskless or ILIO Persistent. These are commands that I personally use so modify as needed.

  1. ILIO appliance snapclone info assuming disk 1 is snapclone:
    dstat -D sdb -cdln --disk-util
  2. ILIO appliance snapclone partition info:
    dstat -D sdb2 -dsr --disk-util
  3. ILIO VM total disk usage:
    df -h /exports/ILIO_VirtualDesktops
  4. ILIO appliance VM info (network, disk, resource with resource limit on nfs stack on dom0):
    dstat -D dm-0 -ndr --nfsd3
  5. ILIO appliance VM info (time, load, network, cpu, disk, resource):
    dstat -D dm-0 -tlncdr --nfsd3
  6. ILIO appliance OS info assuming it is disk 0 (network, disk, resource on disk):
    dstat -D sda -ndr --disk-util
  7. ILIO appliance VM snapclone info assuming it is disk 1 (network, disk, resource on disk):
    dstat -D sdb -ndr --disk-util

 

PowerShell Oneliners

Just open a PowerShell window and drop each line in to get a result.

This first set of one liners returns common redirected folders. If you use AppSense for example, you will likely be redirecting these folders to a file share for each user. You can use these one liners as a base for advanced scripts on reporting redirected user folders for all your users:

  1. Display the redirected Desktop folder:
    [Environment]::GetFolderPath("Desktop")
  2. Display the redirected My Documents folder:
    [Environment]::GetFolderPath("MyDocuments")
  3. Display the redirected My Music folder:
    [Environment]::GetFolderPath("MyMusic")
  4. Display the redirected My Pictures folder:
    [Environment]::GetFolderPath("MyPictures")
  5. Display the redirected My Videos folder:
    [Environment]::GetFolderPath("MyVideos")

 

AutoHotkey Scripts

  1. Left click mouse every 2 seconds in a loop. Good foundation to do more advanced stuff with.

 

3 Comments

  1. Carlos Valdes

    December 31, 2016 at 5:14 AM

    Thanks Jason, this is really good. Looking forward to work with you.

  2. Pingback: EUC Weekly Digest – Februrary 18, 2017 – Carl Stalhood

  3. Jochen Hoffmann

    February 20, 2017 at 4:08 AM

    Jason, great work as usual – thanks for sharing.

Leave a Reply

Your email address will not be published. Required fields are marked *