vSphere 5.5 esxcli namespace updates

This is your 10,000 ft view of the updates to the vsphere 5.5 esxcli.  I do plan to dive in and explore the new additions in more detail at a later date.  This round I just want to provide a taste of what is new.  

For a second time in a row, and much to my delight :D, we see an emphasis on network! (esxcli 5.0 to 5.1 provided a similar treat).  There are changes aligning with some of the new features introduced in vsphere 5.5.  There’s also a significant namespace addition for VSAN.  Duncan Epping provides a great introduction to VSAN here.  Exciting times indeed.  

Without further delay…

Counts of esxcli 5.5 updates

  new commands:  74
      by namespace
      device (2)  graphics (2)  network (26)  sched (1)  storage (11)  system (8) vsan (23) 

  commands removed: 2  (vxlan network mapping replaced with arp/mac/mtep output)
  commands modified: 3  (LACP command cleanup)
  new namespaces: 3 (device, graphics, vsan)


For those who aren’t satisfied with counts, here’s the full delta of 5.1 and 5.5 below.  Enjoy!



## esxcli.new
device.alias get 
device.alias list 
graphics.device list 
graphics.vm list 
network.ip.neighbor remove 
network.ip.netstack add 
network.ip.netstack get 
network.ip.netstack list 
network.ip.netstack remove 
network.ip.netstack set 
network.nic.coalesce get 
network.nic.coalesce set 
network.nic.cso get 
network.nic.cso set 
network.nic.eeprom change 
network.nic.eeprom dump 
network.nic.negotiate restart 
network.nic.register dump 
network.nic.selftest run 
network.nic.sg get 
network.nic.sg set 
network.nic.tso get 
network.nic.tso set 
network.sriovnic.vf stats 
network.vswitch.dvs.vmware.lacp.timeout set
network.vswitch.dvs.vmware.vxlan get 
network.vswitch.dvs.vmware.vxlan.network.arp list 
network.vswitch.dvs.vmware.vxlan.network.arp reset
network.vswitch.dvs.vmware.vxlan.network.mac list 
network.vswitch.dvs.vmware.vxlan.network.mac reset 
network.vswitch.dvs.vmware.vxlan.network.mtep list 
sched.reliablemem get 
storage.nfs.param get 
storage.nfs.param set 
storage.vflash.cache get 
storage.vflash.cache list 
storage.vflash.cache.stats get 
storage.vflash.cache.stats reset 
storage.vflash.device list 
storage.vflash.module get 
storage.vflash.module list 
storage.vflash.module.stats get 
storage.vmfs unmap 
system.coredump.file add 
system.coredump.file get 
system.coredump.file list 
system.coredump.file remove 
system.coredump.file set 
system.security.certificatestore add 
system.security.certificatestore list 
system.security.certificatestore remove 
vsan.cluster get 
vsan.cluster join 
vsan.cluster leave 
vsan.cluster restore 
vsan.datastore.name get 
vsan.datastore.name set 
vsan.maintenancemode cancel 
vsan.network clear 
vsan.network.ipv4 add 
vsan.network.ipv4 remove 
vsan.network.ipv4 set 
vsan.network list 
vsan.network remove 
vsan.network restore 
vsan.policy cleardefault
vsan.policy getdefault 
vsan.policy setdefault 
vsan.storage add 
vsan.storage.automode get 
vsan.storage.automode set 
vsan.storage list 
vsan.storage remove 
vsan.trace set

## esxcli.removed
network.vswitch.dvs.vmware.vxlan.network.mapping list 
network.vswitch.dvs.vmware.vxlan.network.mapping reset

network.vswitch.dvs.vmware.lacp.get config (5.1)
network.vswitch.dvs.vmware.lacp.get stats (5.1)
network.vswitch.dvs.vmware.lacp.get status (5.1)

network.vswitch.dvs.vmware.lacp.config get (5.5)
network.vswitch.dvs.vmware.lacp.stats get (5.5)
network.vswitch.dvs.vmware.lacp.status get (5.5)


vCNS 5.1.x SSL VPN-Plus Client Proxy Settings on Mac won’t stick

Anyone out there attempting to run the SSL VPN-Plus Mac client through a proxy will quickly notice that the settings do not seem to stick.   You check the box to enable proxy setting, add the configuration and click Ok.


All seems well (until you try to connect).  When the settings are applied correctly, the client will reflect that the connection attempt is routed via proxy.


When you open up the proxy settings again, you see the box unchecked and the proxy configuration undefined.  This is a known issue caused by a missing directory where the proxy configuration is written does not get added during install.  This is slated to be fixed in a future release.

As a workaround, you can add the directory manually.  Exit the SSL VPN Mac client, open up the terminal and add the directory:  sudo mkdir -m 777 /opt/sslvpn-plus/naclient/users_dat

Problem solved.  Open up the client, configure the proxy, and connect.

*Credit for the find to N.Albright for quickly finding the problem with dtruss*.


vCNS 5.1.x SSLVPN Gateway IP Change – How to update the IP existing naclient connection profile.

When your SSL VPN-Plus Gateway Interface has its IP changed all new client  attempts to connect will be greeted with this error:

From the client side you have two options.  The first is to connect to the gateway on its new IP, download and run installer from the gateway.  I’ve had mixed results with this method on Mac OS X 10.8.2. I’ve had the best results (non-failed installations) if I’ve freshly rebooted my laptop and have erased all previously downloaded mac_phat client files.

The second option is to update the existing client configurations.  On Windows clients – you can update the windows naclient IPs via the Registry Editor by editing the GatewayList registry key.  Location can slightly change depending on your OS installation.

HKLM\SOFTWARE\VMware, Inc.\SSL VPN-Plus Client\Connection#
HKLM\SOFTWARE\Wow6432Node\VMware,Inc.\SSL VPN-Plus Client\Connection #

As a bonus you can give your name your connections to suit your needs by modifying the ConnectionAlias key (this will change the connection name in the naclient Network dropdown).

If you need to make the update to a Mac OS X, open Terminal, change your directory to /opt/sslvpn-plus/naclient/

Modify naclient.conf

The format is
ClientAlias  GatewayIP  256 

Just make the update to your GatewayIP (and ClientAlias if desired).  Make sure the 256 doesn’t get erased.  Write & Quit, then you are set.

– Gabe


Troubleshooting Invalid Virtual Machines

##Troubleshooting Invalid Virtual Machines##

Virtual machines can show up as invalid if the datastore UUID changed due to resignaturing

*) Try to remove/re-add the VM to vCenter inventory, VM needs to be powered off

Identify VM Host from VM Summary info in vCenter.

Note the path/datastore location using the following commands:

– esxi 4.1 ~# vmware-cmd -l !–gets the full path to vms
– esxi 5.x ~# vim-cmd vmsvc/getallvms !– Note [VMID] / Datastore Volume Name
~# esxcli vm process list !– Note the full path

Identifying vm disk (mapping the full path above to a datastore volume name)

– esxi 4.1 ~# esxcfg-scsidevs -m
– esxi 5.x ~# esxcli storage filesystem list [map path to datastore volume name]

To Remove the VM from Inventory:
– Rick click VM, Remove from Inventory (DO NOT CHOOSE DELETE FROM DISK)
– Browse the datasore Identified above, to the VM directory, right click the .vmx file,
choose Add to Inventory.

*) Determine the sate of the VM

esxi 4.1
~# vmware-cmd <path.vmx> getstate
esxi 5.x
~# vim-cmd vmsvc/power.getstate [VMID]

Interpreting Output

if state is on !– vCenter may not be communicating with the host properly
if state is off !– the ESXi host may be unaware it is still running the virtual machine.

*) If you want to get VM world ID and turn off while debugging

~# vm-support -x ! — get world ID for affected VM.
~# vm-support -X <WorldID> !– Can take several minutes (up to 30 minutes)

*) Powering off the VM gracefully !– works if the VM process is running

esxi 4.1
~# vmware-cmd <path.vmx> stop
~# vmware-cmd <path.vmx> getstate
esxi 5.x
~# vim-vmd vmsvc/power.shutdown [VMID]
~# vim-cmd vmsvc/power.getstate [VMID]

*) Hard Powering off the VM

esxi 4.1
~# vmware-cmd <path.vmx> stop hard !– ungraceful hard shut down of the virtual machine
~# vmware-cmd <path.vmx> getstate

esxi 5.x
~# vim-vmd vmsvc/power.off [VMID]

*) To kill a virtual machine (works in 4.1 and 5.x)
c , for CPU
f , to display fields
c , to add the column for the Leader World ID (LWID)
Identify VM by it’s LWID
k , to bring up the “World to Kill” prompt
type the LWID, press Enter,


vSphere 5.0x ESXi hosts crash when using Trend Micro Deep Security

The environment in question is running ESXi 5.0; using Deep Security(a vShield Endpoint API product by trend micro)… not very sure how common this implementation is.

Blades were crashing intermittently every few days with this resultant PSOD :

The backtrace points to a known symptom seen after installing Trend Micro Deep Security (reference kb.vmware.com/kb/2012584 )

The KB above mentions that we should expect to see dvifilter-dsa errors in the vmkernel log;

Expected entries were found in the log:
2012-08-07T23:25:31.107Z cpu16:771421)dvfilter-dsa: tb_trace_write_formatted:105: alloc_guest:214 guest alloc guid: 420449f6-9a18-8fc5-b116-a32a4c45afe4, domid: 3999995

VMware points to trend micro KB 1060125 for the solution. In a nutshell you have to disable the timer setting causing the problem.


1. From the ESXi console, execute this command to find out the value that is configured for the Filter Driver heap memory size:

% esxcfg-module -g dvfilter-dsa

If the value for the “DSAFILTER_HEAP_MAX_SIZE” is adjusted from its default value then the outcome will be similar to:
dvfilter-dsa enabled = 1 options = ‘DSAFILTER_HEAP_MAX_SIZE=134217728’

2. Use this command to disable timer and preserve the configured value for the DSAFILTER_HEAP_MAX_SIZE:
% esxcfg-module -s “DSAFILTER_HEAP_MAX_SIZE= 134217728 DSAFILTER_MOD_TIMER_ENABLED=0” dvfilter-dsa
Set the DSAFILTER_HEAP_MAX_SIZE to the value that was observed after running the “esxcfg-module -g dvfilter-dsa” command.

If the value for the “DSAFILTER_HEAP_MAX_SIZE” is not changed from its default value then the outcome will be similar to:
dvfilter-dsa enabled = 1 options = ‘ ‘
In this case you can use the following command to disable timers:
% esxcfg-module -s “DSAFILTER_MOD_TIMER_ENABLED=0” dvfilter-dsa


3. Verify if the settings were successfully applied by executing this command:
% esxcfg-module -g dvfilter-dsa
4. Reboot the ESXi server for the changes to take effect.

The setting will not take effect until the driver is reloaded. Reloading will require a reboot (best option) of ESXi or unloading/loading of the driver.

This workaround is not needed if the environment is running Trend Micro Deep Security 7.5 SP4, 8.0 SP1 and later.

Vblock Troubleshooting (Nexus 1000v – Lost communications on properly configured VEM)

• Conditions:

In the configuration nothing has changed, the communications worked previously.  The affected VLAN(s) are configured and allowed correctly on the Nexus 1000v and on the upstream device.  This could mean you are affected by Cisco Bug CSCtz01683.  

Possible triggers:  
– Upgrade to SV1(4a)
– First time channel config is applied to an uplink port profile
– ESX host was rebooted.
– vem restart

First verify the affected VLANs against the VEM:

 n1kv# module vem {VEM#} execute vemcmd show port vlans

 If you are hitting the bug, the affected Vlans will be missing from the uplink interfaces.  

•  Workaround:

To quickly remediate, have the customer migrate the VMs off the host, and run vem restart against that host.

 hotswap.sh -u; sleep 3; hotswap.sh -l; sleep 5; vem restart

If the vem restart does not resolve, reboot the host.

• Permanent Fix:

Upgrades from SV1(4a) to a future certified release.