Vblock Troubleshooting (Nexus 1000v – Lost communications on properly configured VEM)

• Conditions:

In the configuration nothing has changed, the communications worked previously.  The affected VLAN(s) are configured and allowed correctly on the Nexus 1000v and on the upstream device.  This could mean you are affected by Cisco Bug CSCtz01683.  

Possible triggers:  
– Upgrade to SV1(4a)
– First time channel config is applied to an uplink port profile
– ESX host was rebooted.
– vem restart

First verify the affected VLANs against the VEM:

 n1kv# module vem {VEM#} execute vemcmd show port vlans

 If you are hitting the bug, the affected Vlans will be missing from the uplink interfaces.  

•  Workaround:

To quickly remediate, have the customer migrate the VMs off the host, and run vem restart against that host.

 hotswap.sh -u; sleep 3; hotswap.sh -l; sleep 5; vem restart

If the vem restart does not resolve, reboot the host.

• Permanent Fix:

Upgrades from SV1(4a) to a future certified release.