Monday, November 9, 2015

Do you like PowerCLI and NSX? Add Layer 3 VDR/LDR capabilities to esxcli / PowerCLI


Long time no write.  I've been working with a quite a few customers recently and have been working on developing PowerCLI checks so that customers do not have to login to SSH terminals for every host to run NSX Sanity Checks.  Remember to "Connect-VIServer" first!

PowerCLI version of "net-vdl2 -l"

netvdl_query.ps1

This will provide Format-Table outputs of Esxcli responses from all connected hosts in the targeted clusters.  It lets you verify your VNI's are connected, online, and that controller connections are proper.

Namespaces queried:
$esxcli.network.vswitch.dvs.vmware.vxlan.get()
$esxcli.network.vswitch.dvs.vmware.vxlan.list()
$esxcli.network.vswitch.dvs.vmware.vxlan.network.list(vxlanvds)
$esxcli.network.ip.interface.list("vxlan")
$esxcli.network.ip.interface.ipv4.get()

PowerCLI C:\pcli\nsx> .\netvdl2_query.ps1 Physical
hivemind.united.earth
VXLAN Global States:

ControlplaneOutOfSync UDPport
--------------------- -------
No                    8472


VXLAN VDS:

GatewayIP    GatewayMAC        MTU  NetworkCount SegmentID    VDSID                                           VDSName   VmknicCount
---------    ----------        ---  ------------ ---------    -----                                           -------   -----------
192.168.24.1 ff:ff:ff:ff:ff:ff 9000 9            192.168.24.0 28 2c 38 50 93 cc 28 37-8e 56 33 cd 7c 1a ea cf nsxswitch 2


VTEP VMKNIC:

Enabled ExternalID MACAddress        MTU  Name NetstackInstance OpaqueNetworkID OpaqueNetworkType PortID   Portgroup
------- ---------- ----------        ---  ---- ---------------- --------------- ----------------- ------   ---------
true    N/A        00:50:56:6f:ca:99 9000 vmk1 vxlan            N/A             N/A               67108979 N/A
true    N/A        00:50:56:68:17:93 9000 vmk2 vxlan            N/A             N/A               67108980 N/A



AddressType DHCPDNS IPv4Address  IPv4Broadcast  IPv4Netmask   Name
----------- ------- -----------  -------------  -----------   ----
STATIC      false   192.168.24.4 192.168.24.255 255.255.255.0 vmk1
STATIC      false   192.168.24.7 192.168.24.255 255.255.255.0 vmk2


VXLAN VNI:

ARPEntryCount ControlPlane                        ControllerConnection MACEntryCount MulticastIP               PortCount VXLANID
------------- ------------                        -------------------- ------------- -----------               --------- -------
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    2             N/A (headend replication) 1         5001
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    2             N/A (headend replication) 2         5000
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    2             N/A (headend replication) 2         5006
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    2             N/A (headend replication) 1         5007
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    0             N/A (headend replication) 1         5002
0             Disabled                            0.0.0.0 (down)       0             0.0.0.0                   1         5004
0             Disabled                            0.0.0.0 (down)       0             0.0.0.0                   1         5005
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    2             N/A (headend replication) 2         5003
0             Enabled (multicast proxy,ARP proxy) 192.168.1.90 (up)    1             N/A (headend replication) 2         5009


Next up is a PowerCLI check to make sure a vdrPort exists on your hosts, Please note this script defaults to using "nsxswitch" as the DV Switch Name to Check, and VNI 5000, you can alter the script or specify the switch name and VNI manually as parameters.  This script is useful in making sure your hosts haven't encountered the "Would Block" issue which can affect both VTEP, and LIF creation.

This verifies that a VDR instance has at least been created and was my initial work around to the net-vdr tool not being exposed by esxcli.  It specifically looks for the "vdrPort" instance on your DV Switch.

vdrPort_check.ps1

PowerCLI C:\pcli\nsx> .\vdrPort_check.ps1 Physical nsxswitch 5001
hivemind.united.earth
1 vdrPort was found
hivemind2.united.earth
1 vdrPort was found
hivemind3.united.earth
1 vdrPort was found



Now to the bread and butter of the post.  ESXCLI-NETVDR, have you wanted insight into your LDR/VDR instances on your hosts via PowerCLI but been left hanging because the Layer 3 information is no where to be found?   The following VIB will let you get what you need!  This allows access to a specific subset of net-vdr commands within ESXCLI, and more important extended into PowerCLI via the Get-ESXCLI object.  Unfortunately at this time it is only CommunitySupported. 

esxcli-netvdr.vib

It creates the following namespaces under ESXCLI:

network.vdr
network.vdr.nbr
network.vdr.route
network.vdr.lif


The following commands:

network.vdr.list
network.vdr.lif.list
network.vdr.nbr.clear
network.vdr.nbr.list
network.vdr.route.list



Here it is in action from an SSH session, but this is not what it's for, it's for PowerCLI

[root@hivemind:~] esxcli network vdr list
default+edge-15
   VdrName: default+edge-15
   VdrId: 0x00001389
   NumberOfLifs: 1
   NumberOfRoutes: 1
   State: Enabled
   ControllerIP: 192.168.1.90
   ControlPlaneIP: 192.168.1.20
   ControlPlaneActive: Yes
   NumUniqueNexthops: 0
   GenerationNumber: 0
   EdgeActive: Yes

default+edge-4
   VdrName: default+edge-4
   VdrId: 0x00001388
   NumberOfLifs: 7
   NumberOfRoutes: 10
   State: Enabled
   ControllerIP: 192.168.1.90
   ControlPlaneIP: 192.168.1.20
   ControlPlaneActive: Yes
   NumUniqueNexthops: 1
   GenerationNumber: 0
   EdgeActive: No



So how can I use these mystical commands via PowerCLI, once the VIB is installed make sure to restart management agents, or reboot your host.  The ESXCLI namespaces won't be available until hostd is restarted.  In all my examples below I will query my LDR instance "default+edge-4"

Start off by connecting to your vCenter "Connect-VIServer"
Then grab an ESXCLI Object from a host.

PowerCLI C:\pcli\nsx> Connect-VIServer cosmos

Name                           Port  User
----                           ----  ----
cosmos                         443   NADESICO\dieth


PowerCLI C:\pcli\nsx> $esxcli = Get-ESXCLI -VMHost hivemind.united.earth
PowerCLI C:\pcli\nsx> $esxcli.network.vdr.list()


ControlPlaneActive : Yes
ControlPlaneIP     : 192.168.1.20
ControllerIP       : 192.168.1.90
EdgeActive         : Yes
GenerationNumber   : 0
NumUniqueNexthops  : 0
NumberOfLifs       : 1
NumberOfRoutes     : 1
State              : Enabled
VdrId              : 0x00001389
VdrName            : default+edge-15

ControlPlaneActive : Yes
ControlPlaneIP     : 192.168.1.20
ControllerIP       : 192.168.1.90
EdgeActive         : No
GenerationNumber   : 0
NumUniqueNexthops  : 1
NumberOfLifs       : 7
NumberOfRoutes     : 10
State              : Enabled
VdrId              : 0x00001388
VdrName            : default+edge-4



Want a list of just your VdrNames?

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.list() | Select-Object VdrName

VdrName
-------
default+edge-15
default+edge-4


How about all the connected logical interfaces?  (I've selected a subset of the columns to fit into the blog)

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.lif.list("default+edge-4") | FT -Property LIFName,ID,ConnectedDvs,State,Mode -AutoSize

LIFName      ID         ConnectedDvs State   Mode
-------      --         ------------ -----   ----
13880000000f Vxlan:5000 nsxswitch    Enabled Routing, Distributed, Internal
138800000002 Vxlan:5001 nsxswitch    Enabled Routing, Distributed, Uplink
13880000000c Vxlan:5005 nsxswitch    Enabled Routing, Distributed, Internal
13880000000b Vxlan:5004 nsxswitch    Enabled Routing, Distributed, Internal
13880000000a Vxlan:5002 nsxswitch    Enabled Routing, Distributed, Internal
13880000000e Vxlan:5007 nsxswitch    Enabled Routing, Distributed, Internal
13880000000d Vxlan:5006 nsxswitch    Enabled Routing, Distributed, Internal


How about all the neighbor information for a specific LIF? (Again I've selected a subset)

 PowerCLI C:\pcli\nsx> $esxcli.network.vdr.nbr.list("138800000002","default+edge-4") | FT -Property Network,Interface,Mac,SrcPort -AutoSize

Network     Interface    Mac               SrcPort
-------     ---------    ---               -------
172.16.10.2 138800000002 02:50:56:56:44:52 0
172.16.10.1 138800000002 00:50:56:b8:1a:d0 67108868


What if I want all LIFs neighbor information?

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.lif.list("default+edge-4") | %{ $esxcli.network.vdr.nbr.list($_.LIFName,"default+edge-4") } | FT  Network,Interface,Mac,SrcPort -AutoSize

Network      Interface    Mac               SrcPort
-------      ---------    ---               -------
172.16.254.3 13880000000f 00:50:56:b8:68:36 67108998
172.16.254.1 13880000000f 02:50:56:56:44:52 0
172.16.10.2  138800000002 02:50:56:56:44:52 0
172.16.10.1  138800000002 00:50:56:b8:1a:d0 67108868
172.16.5.1   13880000000c 02:50:56:56:44:52 0
172.16.4.1   13880000000b 02:50:56:56:44:52 0
172.16.12.1  13880000000a 02:50:56:56:44:52 0
172.16.19.2  13880000000e 00:50:56:b8:48:48 67108868
172.16.19.1  13880000000e 02:50:56:56:44:52 0
172.16.8.1   13880000000d 02:50:56:56:44:52 0
172.16.8.2   13880000000d 00:50:56:b8:33:92 67108984



Finally what if I want to clear the neighbor information?

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.lif.list("default+edge-4") | %{ $_.LIFName ; $esxcli.network.vdr.nbr.clear($_.LIFName,"default+edge-4") }
13880000000f
Complete
138800000002
Complete
13880000000c
Complete
13880000000b
Complete
13880000000a
Complete
13880000000e
Complete
13880000000d
Complete


Alternatively to clear neighbor information on a per LIF basis:

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.nbr.clear("13880000000f","default+edge-4")
Complete


Last but not least list all routes available via from your specific LDR/VDR instances:

PowerCLI C:\pcli\nsx> $esxcli.network.vdr.route.list("default+edge-4") | FT -AutoSize

Destination  Flags Gateway     GenMask       Interface    Origin Ref Uptime
-----------  ----- -------     -------       ---------    ------ --- ------
0.0.0.0      UG    172.16.10.1 0.0.0.0       138800000002 AUTO   1   551207
172.16.4.0   UCI   0.0.0.0     255.255.255.0 13880000000b MANUAL 1   1768288
172.16.5.0   UCI   0.0.0.0     255.255.255.0 13880000000c MANUAL 1   1768288
172.16.7.0   UG    172.16.10.1 255.255.255.0 138800000002 AUTO   1   551207
172.16.8.0   UCI   0.0.0.0     255.255.255.0 13880000000d MANUAL 1   1768288
172.16.10.0  UCI   0.0.0.0     255.255.255.0 138800000002 MANUAL 1   1768288
172.16.12.0  UCI   0.0.0.0     255.255.255.0 13880000000a MANUAL 1   1768288
172.16.19.0  UCI   0.0.0.0     255.255.255.0 13880000000e MANUAL 11  1768288
172.16.254.0 UCI   0.0.0.0     255.255.255.0 13880000000f MANUAL 1   1768288
192.168.1.0  UG    172.16.10.1 255.255.255.0 138800000002 AUTO   6   551207