JasonSamuel.com

Had a situation where I needed to reset every VM in one environment to Unlimited that had previously had a an explicitly set memory limit. Here is a nice VMware PowerCLI script that will reconfigure all the affected VMs one by one automatically for you:

get-vm|get-vmresourceconfiguration|where-object{$_.memlimitmb -ne '-1'}|set-vmresourceconfiguration -memlimitmb $null

Wonderful little vSphere bug we came across today. Let’s say you have a host with the following VMKernel ports configured on the vSwitch:

-vmk1 = vMotion
-vmk0 = Management Network

When you create a host profile from the host, then apply that profile to a new host you are adding, the VMKernel ports get swapped! They will be:

-vmk1 = Management Network
-vmk0 = vMotion

My co-worker came up with a good workaround. Put your affected host into maintenance mode, then go edit the Host Profile you had applied. It appears the kernel ports are created in the order in the Host Profile. So you just change the order from:

-vMotion
-Management Network

to:

-Management Network
-vMotion

You do this by renaming “vMotion” to something like “vMotion2MN”. Then change the name of “Management Network” to “vMotion” and change “vMotion2MN” to “Management Network”. From here you need to make changes to each port group which are:

-make sure the NICs are ordered correctly
-specifying the correct service for the port group
-making sure the IP is set to explicit

So for the vMotion port group, order the NIC:

Choose vmotion for the service:

Set the explicit IP option:

And for the Management Group, order the NIC:

Choose management for the service:

And set the IP to be explicit:

Make sure to press the OK button in the lower right button after every page or when you navigate to another page, it will lose whatever changes you made. Save the profile and re-apply it to your new host. Now it will recreated the VMKernel ports in the correct order.

I’ve had this happen several times when an ESX host is stuck thinking it is going into maintenance mode from previous maintenance. You get an error saying it cannot complete the migration and “The operation is not allowed in the current state”. Very cryptic error. Just restart the management agents on the ESX host by SSHing in and typing:

service mgmt-vmware restart

hit enter, then:

service vmware-vpxa restart

and hit enter again. You should see the services restart with an "[ OK ]" after each line.

Try to vMotion again now. If you still get an error, disconnect the ESX host from Virtual Center and then reconnect it. Try to vMotion again and it should work. You can also try restarting VC services too. If you have a better solution, let me know. I know it is a bit tedious to go through this process but it does work.

PowerPath/VE replaces the native mulipathing plugin which gives you incredible performance gains when managing your EMC Clariion SANs. You have to push this plugin to all your vSphere/ESX hosts to take advantage of it. Using VMware Update Manager is the easiest solution but I ran into a problem. Adding a 3rd party extension requires adding a patch repository in VUM and it only accepts web URLs. So I would have to setup a dedicated web server just for this patch repository which is ridiculous. Instead I decided to leverage my Virtual Center server itself since I knew it uses a Tomcat installation to display the vCenter web GUI and webservices. Some quick Googling and I found Joshua Townsend had the exact same idea and wrote a nice guide to do this already:

http://vmtoday.com/2010/02/installing-powerpathve-using-vmware-update-manager/

Once I set it up the way Joshua describes and pointed VUM to the patch repository URL hosted by vCenter, it worked perfectly. You will notice in my screenshot below Connectivity Status says “Connected” now.

IP Address conflict? Have you narrowed the MAC down to being a VM in one of your ESX/vSphere clusters? Well use vSphere PowerCLI to figure out what device and adapter is causing the conflict.

Example, if your MAC is 005056b60b13:

Get-vm | Select Name, @{N="Network";E={$_ | Get-networkAdapter | ? {$_.macaddress -eq "00:50:56:b6:0b:13"}}} |Where {$_.Network-ne ""}

And it will output the VM name and the Network Adapter causing the conflict.

A co-worker and myself were asked to create a VMware vSphere (ESX) template for Server 2008 R2 and it’s not as easy as you might think. Over the course of a day and half and through much research and trial and error, we have come up with the perfect template for our organization. It’s a perfect base template for us but depending on where you intend to deploy it and your own organization’s requirements, you may have to edit it a bit. We intend to tweak the template even further and possibly even create separate images depending on if the VM will be deployed internally or externally. Please do comment if you have any suggestions or think we may have missed something.

Building a VMware template for Server 2008 vs. Server 2008 R2 has some differences. This blog post will cover everything we did to successfully get it working and a detailed explanation of why we chose certain settings. Big thanks to to Jeremy Waldrop and his blog post that described setting up a template for 2008 which helped us quite a bit in our research to create the perfect template for 2008 R2.

OS Used: Windows Server 2008 R2 Standard (x64) Volume License Edition

VM Hardware config:

• Single vCPU
• 4 GB RAM
• 40 GB Primary Hard Drive (for Operating System)
• 10 GB Secondary Hard Drive (for Page File
• LSI Logic SAS SCSI Controller
• VMXNet 3 Network Adapter

I know most organizations using Server 2003 have always done a 20 GB C: partition. With 2008, I suggest going to 40 GB OS drives as a standard. Server 2008 (especially x64 versions) requires more space. Out of the box 2008 R2 x64 takes up 10 GB. Also Server 2008 has a component store (c:\windows\winsxs) which is very large. This is because 2008 no longer uses i386, everything is stored locally already in this component store folder. Remember, Server 2008 and any future MS product is all about componentization!! When you install a component from this store, my understanding is that it is “projected” to the OS. So basically Windows 2008 installed components run from this component store essentially. As the system receives updates over time, expect this directory to grow even larger since it never deletes old stuff. Think of the different versions of kind of stacking on each other. I believe with each service pack there is a tool to uninstall components that are no longer necessary or superseded. So stick with a 40 GB OS partition and you should be fine for a long time. You will also notice we have a 10 GB secondary drive for a page file. We’ll get to that later in the article.

Now on to the actual build:

1. First create a new VM
   • Select Custom Configuration
   • Enter VM Name and Inventory Location
   • Select Datastore
   • Select Virtual Machine Version: 7
   • Select “Microsoft Windows Server 2008 R2 (64-bit)” as OS Version
   • # of Virtual Processors: 1
   • Amount of RAM: 4GB
   • Network
     • # of NICs: 1
     • Adapter Type: VMXNET 3
     • Select “Connect at Power On
   • SCSI Controller: LSI Logic SAS
   • Create New Virtual Disk: 40GB
   • Advanced Options: No Change
2. Now prepare the virtual hardware:
   • Edit VM Settings > Options > General Section > Uncheck “Enable logging”
   • Boot Options > Check box to force going into the BIOS on next boot
   • Power on the VM (will go directly to BIOS) > Advanced > I/O Device Configuration:
     • Disable Serial port A
     • Disable Serial port B
     • Disable Parallel port
   • Exit and Save
3. OS Installation and Configuration
   • Install Windows 2008 R2 Standard – Full Install
   • After OS install and reboot, change Administrator Password (will prompt)
   • Disconnect Windows 2008 R2 ISO and set device type to Client Device
   • Set Time Zone
   • VMware Tools Install
     • Install VMtools, choose Custom Install Type
     • Disable the “Shared Folders” drive and install Tools ** Note we are disable Shared Folder due to profile loading issues which was documented even back in ESX 3.5 and VMware Tools here on the VMare Communities forum. I have not personally had an issue leaving it enabled but just to be cautious and the fact we don’t use this feature in our organization, we have left it disabled.
     • Set time synching between the VM and ESX host
     • Reboot after Tools Install
   • Network Configuration
     • From Server Manager, select View Network Connections
     • Right click on Local Area Connection and select properties
     • Uninstall QoS Packet Scheduler and both Link-Layer Topologies (Mapper & Responder) ** We don’t do QOS at the server level, our switches do that. Link Layer is not used by us.
     • Uncheck IPv6 and close network connection screens ** We don’t use IPv6 yet so we disabled it for now
   • Server Name
     • From Server Manger select Change System Properties
     • On System Properties screen click Change on Computer Name Tab
     • Set Server Name and restart
   • Windows Updates
     • From Server Manager under Security Information, select Configure Updates
     • Select Let me choose
     • Under Important Updates, select Never check for updates, click ok
     • Start > All Programs > Windows Update > Check for updates and install all Recommended Updates
   • Enable Remote Desktop, choose “Allow connections from computers running any version of Remote Desktop” (2^nd option)
   • Disable Windows Firewall **Not best practice to disable, but my environment requires it
   • From Server Manager, select “Do not show me this console at logon” and close Server Manager
   • Taskbar Changes
     • Right click 3^rd icon from Start Button (Windows Explorer) and select “Unpin this program from taskbar”
     • Right click 2^nd icon from Start Button (Windows PowerShell) and select “Unpin this program from taskbar”
     • Right click Taskbar and choose Properties and choose Customize under Notification Area
     • Select “Turn system icons on or off”, and turn Volume Off, click Ok
   • System Performance
     • From Server Manager select Change System Properties
     • Select Advanced Tab > Settings and choose “Adjust for best performance”
   • Folder and Search Options
     • Open “Computer” > Select Organize > Choose Folder and search options
     • Under View Tab
       • Select “Show hidden files, folders and drives”
       • Uncheck “Hide extensions for known file types”
   • IE ESC
     • From Security Information Section, select Configure IE ESC
       • Change Administrators to Off and leave Users On ** My reasoning for this is the only “Users” should be service accounts on a server so leaving it On should not matter
   • Change IE Home Page to blank so you don’t get that pesky Internet Exploer Enhanced Security Configuration warning page
   • Under Computer, right click c: and select properties, uncheck “Allow files on this drive to have contents indexed in addition to file properties”
     • Apply changes to c:\ and all subfolder/files
     • Continue/Ignore on Access Denied errors
   • Power Options (from Control Panel)
     • Change option to High Performance
   • Disable Hibernation
     • Command Prompt, enter powercfg.exe –h off
   • Delete the Page file and reboot (so c:\ can be fully defragmented)
   • Run defrag
   • Page File
     • Edit VM Properties
     • Add a 2^nd hard drive (10GB) and change to SCSI (1:0)
     • Run Disk Manager and format as Z:\ drive ** We use Z: as the drive letter so it does not interfere with adding additional drives later on.
     • From Server Manager, select System Properties > Advanced > Performance Settings > Advanced > Virtual Memory Change
       • Assign 1024MB Page file to c:\
       • Assign 5120MB Page file to z:\
   • SNMP
     • Server Manager > Add Features > SNMP
     • Server Manager > Configuration > Services > SNMP > Security
       • Accepted community names – Add your community (as READ ONLY)
       • Accept SNMP Packets from these hosts – Add your hosts (remember to leave localhost in there)
4. Turn VM into a Template
   This procedure will copy the Administrator account profile into the default user profile so that all users that login or that will be created in the future will get the same profile with all the customizations you have done above. When you sysrep a server template and create a new VM from it, a new SID is generated which means a new local Administratior account is created during the sysrep procedure. This means all the customizations you have done will be wiped out above unless you copy all your settings above when you create a new VM. In the past with Server 2003 and even Server 2008, you had the “Copy To” feature to copy a user profile to another. With Server 2008 R2, Microsoft has disabled this feature. It is now done via an unattend.xml file using the “CopyProfile” node. I actually prefer this method now after doing it a few times. This procedure is detailed per the Microsoft KB article http://support.microsoft.com/kb/959753
   • Create unattend.xml in “c:\windows\system32\sysprep” folder as follows. NOTE: Do not copy and paste the text below because WordPress messes up the quotes which will lead to errors during sysprep. Please right click-save as this link and copy and paste from the txt file instead.
     

     
     <?xml version="1.0" encoding="utf-8"?> 
     <unattend xmlns="urn:schemas-microsoft-com:unattend"> 
     <settings pass="specialize"> 
     <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 
     <RegisteredOrganization>Your Organization Name</RegisteredOrganization> 
     <RegisteredOwner>Your Registered Owner</RegisteredOwner> 
     <CopyProfile>true</CopyProfile> 
     </component> 
     </settings> 
     <cpi:offlineImage cpi:source="wim:f:/sources/install.wim#Windows Server 2008 R2 SERVERSTANDARD" xmlns:cpi="urn:schemas-microsoft-com:cpi" /> 
     </unattend>
     

   • At command prompt, type the following command:
     C:\windows\system32\sysprep\sysprep.exe  /generalize /unattend:unattend.xml
   • System Preparation Tool 3.14
     • Choose Enter System OOBE
     • Check Generalize button
     • Leave shutdown option as reboot
     • Click OK
   • Release the IP and turn VM into a template in Virtual Center

I hope this guide we wrote helps someone out there and saves you time. If you have any suggestions or questions, please leave a comment. I can take screenshots of any parts above you are confused on. I am also interested in knowing about more performance related customizations you guys might use we can add to your template so if you have any, please let me know.

Have you ever P2Ved a server to an ESX cluster or converted a VMware Workstation VM into a VMware ESX 3.5 or vSphere 4.0 VM and come across this horrible Found New Hardware message that won’t go away.  Here’s a screenshot of a Windows 2000 Advanced Server that I had to fix and P2V after a catastrophic disaster (RAID failure) displaying this message:

Yes, it’s annoying and it seems like it will never go away.  You keep click on install and it keeps detecting after sucessfully installing as if it’s in a loop.  But the reality of it is that it’s not in a loop.  As silly as it sounds, you have to install it 32 times and it will finally go away!

The reason for this is because when you update the VM’s virtual hardware (specifically version 7 VMs from my experience), it comes with 32 separate PCI-to-PCI bridge devices that Windows detects and wants to install.  The only workaround I have found so far is to just sit there and install all 32 manually.  Or you can try editing your VMX and set:

pciBridge.present = False

which should work though I’ve never tried it personally.  Don’t want to potentially break something when it takes all of 2 minutes to install all 32 bridges.  Give it a shot and let me know if that fixes it for you guys.

VMware vSphere PowerCLI is an extremely powerful command line utility you can use to automate every aspect of your ESX environment.  We’re talking every level from VMs, the host OS, storage, networking, licensing, and more!  Basically its a ton of PowerShell cmdlets specifically for ESX that sit on top of Windows PowerShell.

You can download it here if you never played with it before:

http://communities.vmware.com/community/vmtn/vsphere/automationtools/windows_toolkit?view=overview

They have very good documentation and sample scripts to try out at the link above.  Now the fun part.  What’s the first script you should try out?  My advice is vCheck by Alan Renouf.

PowerCLI vCheck (Daily Report) – http://www.virtu-al.net/2009/11/04/vcheck-daily-report-v3/ – Without a doubt one of my favorite PowerCLI scripts.  Every morning when you arrive to the office, you can have an email report waiting in your Outlook Inbox with the status of your entire ESX environment.  Every aspect of it.  From old snapshots to resource utilization on your clusters, it’s all there.  I first started using it at version 1 but version 2 is even slicker!  If you are an ESX admin, you NEED to be using this script!  The lastest version 3 came out the other day and I haven’t used it in a production environment yet but you can download it at the link above.  Al has gone back to calling it vCheck by the way instead of Daily Report.  I’ll be tweaking V3 at home this weekend hopefully if I have the time and test it out.

You can see more scripts from Alan on his blog:

http://www.virtu-al.net/script-list/

I could list them all here with descriptions but he’s done a good job of keeping everything organized in the link above. Try them out!  Let me know if you have any favorite scripts you use.  I’m always looking for new scripts to make life easier so share please.

DISCLAIMER:  Don’t even think about trying this in a production enterprise environment or you will get laughed out of the building.  Don’t say I didn’t warn you!  This is meant for test labs and small businesses.

With that said, if you’re running ESXi 4.0 at home or at a small business and want image level backups at a pricepoint (i.e. FREE), then this is how you do it.  You used to be able to use products from 3rd parties with ESXi 3.5 such as Veeam Backup that would allow you to do image based backups but once ESXi 4.0 (vSphere) came out, VMware decided they wanted to charge small and medium size business for advanced features.  Previously this was an untapped market for VMware but now they want a piece of the pie.  So they went to these 3rd parties that were offering image level backup solutions for ESXi 3.5 and said not to support it anymore with ESXi 4.0.  Here is a press release from Veeam regarding this here:

http://www.veeam.com/news/veeam-software-offers-new-essentials-bundle-acceleration-kits-for-vmware-vsphere4.html

As a longtime Premier Technology Alliance Partner and supporter of VMware’s product strategy, Veeam Software takes great pride in creating innovative software products that enhance the customer value of VMware ESX, ESXi, and ESXi Free.  One such example is support for the free edition of VMware ESXi in Veeam Backup and Replication.

Recently, VMware requested that Veeam Software discontinue support for ESXi Free in Veeam Backup and Replication in order to comply with VMware’s updated licensing policy.  In light of VMware’s request, and our close technical partnership, Veeam Backup and Replication will no longer support ESXi Free. We will still continue to offer support for ESXi Free to existing Veeam customers who purchased Backup & Replication prior to version 3.1.

For small or branch office environments, Veeam advises its new customers to buy VMware vSphere Essentials and the Veeam Essentials bundle (announced today).  This combination costs exactly the same amount of money as purchasing 6 sockets of Veeam Backup at regular list price for use with free ESXi, but it is actually better, because with Veeam Essentials at its special discounted price, customers also get Veeam Monitor and Veeam Reporter.

So,

1. ESXi 4.0 (still free) = basic features and an entry into virtualization for small companies or test labs
2. vSphere Essentials ($995) = all the above plus advanced features aimed at small business
3. vSphere Essentials Plus ($2,995) = all the above but for more processors plus more features such as VCB

Here’s a nice product matrix that explains all the differences:

http://www.vmware.com/products/vsphere/buy/editions_comparison.html

I’m can’t be too mad at VMware about this.  They make a great product and want to tackle a new market.  It’s a smart business move for them.  Especially with MS Hyper-V trying to tackle this same market.  Veeam was very innovative in coming up with a great backup solution but I understand they want to maintain a good relationship with VMware and don’t want to step on their toes.

But for those that miss the old days of setting up a quick and easy DR solution for an ESXi host, how can we get the old functionality we were used to back without having to shell out thousands for VMware Consolidated Backup (VCB) or Essentials Plus?  In comes GhettoVCB, an ongoing project on the VMware Community forums:

http://communities.vmware.com/docs/DOC-8760

GhettoVCB is a free alternative for backing up guest images on ESXi 3.5 and ESXi 4.0.  I have to warn you though, it does take some work to get it working properly.  If you don’t want the hassle, I encourage you to buy VCB or Essentials Plus which is definitely worth it and of course comes with support from VMware.  GhettoVCB is a great project and does work but again as I said in the disclaimer at the top of this post, I wouldn’t advise using it in a production environment.  It’s great to play with in a test lab or at a small company though.  If you like playing with things, you will enjoy setting it up.