|
Size: 7176
Comment:
|
← Revision 74 as of 2025-04-02 06:32:20 ⇥
Size: 8914
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| <<TableOfContents()>> | |
| Line 3: | Line 4: |
| == Get all VM power state and more == * esxcli vm process list * for vmid in `vim-cmd vmsvc/getallvms | awk '{ print $1 }' | grep -v Vmid`; do echo $vmid; done | awk 1 ORS=' ' {{{ for vmid in `vim-cmd vmsvc/getallvms | awk '{ print $1 }' | grep -v Vmid`; do vim-cmd vmsvc/get.summary $vmid | egrep -i 'ipAddress|vim.VirtualMachine|powerState| hostname =';done }}} == Rescan storage == Initiate a storage rescan on the ESXi host through the vCenter Server GUI by right click on the ESXi host and select Storage > Rescan Storage) or through the command line by executing this command: {{{ vmkfstools -V }}} == identify DEVICE ERROR CODE == https://www.virten.net/vmware/esxi-scsi-sense-code-decoder/?host=0&device=2&plugin=0&sensekey=5&asc=24&ascq=03&opcode=1a == Beacon Probing Deep-Dive == * https://learnvmware.online/2019/03/25/beacon-probing-deep-dive/ * https://blogs.vmware.com/vsphere/2008/12/using-beaconing-to-detect-link-failures-or-beaconing-demystified.html {{{ With 3 or more uplinks in a team, we can pin point failures of a single uplink. With 2 uplinks in a team, we can detect downstream link failure, but we don't know which one is good and which bad. }}} * ESXi with two NICs If you have three or more NICs, the mechanism is pretty straight-forward, but what happens if you have only two NICs? In this situation, you cannot determinate if the problem is caused by the sending NIC ort he receiving NIC. There is a fallback mechanism that will be used in this situation. Since ESXi does not know which of the NIC is affected (but it knows that one of the NIC is affected), it will simply starts to duplicate the frames on both NICs == iSCSI == {{{ esxcli iscsi adapter discovery sendtarget list }}} |
|
| Line 88: | Line 132: |
| == Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x (2004684) == * https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vsphere.storage.doc/GUID-FF0426FB-57AE-4045-BB8F-565891FDC2AC.html * https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vsphere.storage.doc/GUID-A513D44C-71DE-47ED-B781-327F78659404.html {{{ Permanent Device Loss (PDL): A datastore is shown as unavailable in the Storage view A storage adapter indicates the Operational State of the device as Lost Communication All-Paths-Down (APD): A datastore is shown as unavailable in the Storage view. A storage adapter indicates the Operational State of the device as Dead or Error. }}} {{{ To clean up an unplanned PDL: All running virtual machines from the datastore must be powered off and unregistered from the vCenter Server. From the vSphere Client, go to the Configuration tab of the ESXi host, and click Storage. Right-click the datastore being removed, and click Unmount. The Confirm Datastore Unmount window displays. When the prerequisite criteria have been passed, the OK button appears. If you see this error when unmounting the LUN: Call datastore refresh for object <name_of_LUN> on vCenter server <name_of_vCenter> failed You may have a snapshot LUN presented. To resolve this issue, remove that snapshot LUN on the array side. Perform a rescan on all of the ESXi hosts that had visibility to the LUN. Note: If there are active references to the device or pending I/O, the ESXi host still lists the device after the rescan. Check for virtual machines, templates, ISO images, floppy images, and raw device mappings which may still have an active reference to the device or datastore. If the LUN is still being used and available again, go to each host, right-click the LUN, and click Mount. Note: One possible cause for an unplanned PDL is that the LUN ran out space causing it to become inaccessible }}} == Unable to open MKS: Internal Error” when opening virtual machine console == https://kb.vmware.com/s/article/2116542 http://noor2122.blogspot.com/2015/12/open-vm-console-unable-to-connect-to.html == Failed to log into NFC server == While looking for causes I found these possibilities: I. Not enough space in the datastore I. Port 901/902 not open between the VI Client to the source of the file (and/or VC Server, could not find if that is also a factor) I. DNS configuration for the host servers As a possible solution I also found that restarting the VC Server service could help. * https://community.spiceworks.com/topic/1956130-issues-with-esxi-6-5 == Deprecated VMFS volume(s)found on the host. Please consider .... == services.sh restart == how to upgrade == vim-cmd hostsvc/maintenance_mode_enter esxcli --server=server_name software vib update --depot=/path_to_vib_ZIP/ZIP_file_name.zip vim-cmd hostsvc/maintenance_mode_exit === How to upgrade ESXi 6.0 to ESXi 6.5 via Offline Bundle – The Steps: === {{{ # esxcli software sources profile list -d /vmfs/volumes/NEW-HX-TEST01/HX-Vmware-ESXi-6.5U1-5969303-Cisco-Custom-6.5.1.1-Bundle.zip Name Vendor Acceptance Level ------------------------------------------------- ------ ---------------- Vmware-ESXi-6.5.0-HX-5969303-Custom-Cisco-6.5.1.1 CISCO PartnerSupported [root@BGY-HX-13:/vmfs/volumes/b897e8ce-f4ee56e5] esxcli software profile update -d /vmfs/volumes/NEW-HX-TEST01/HX-Vmware-ESXi-6.5U1-5969303-Cisco-Custom-6.5.1.1-Bundle.zip -p Vmware-ESXi-6.5.0-HX-5969303-Custom-Cisco-6.5.1.1 Update Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true }}} esxcli software profile update -p ESXi-6.5.0-4564106-standard -d /vmfs/volumes/your_datastore/VMware-ESXi-6.5.0-4564106-depot.zip == Interpreting SCSI sense codes in VMware ESXi and ESX == * SCSI events that can trigger ESX server to fail a LUN over to another path https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003433 == vim-cmd how to == {{{ For HBA information only vmware-vim-cmd hostsvc/summary/hba For datastore information only vmware-vim-cmd hostsvc/summary/fsvolume esxcfg-scsidevs -l }}} == show hba controller == * Dell PERC {{{ /opt/lsi/perccli/perccli /c0 show }}} == Find vmx by displayName == {{{ find /vmfs/volumes/BGYBH-HX-PRD/ -iname "*.vmx" -type f -print0 | time xargs -0 egrep -il "displayName.*ehrmaster03" 2>/dev/null }}} |
== ESXi network card driver == {{{ esxcli network nic get -n vmnic0 }}} == scratch warning == * Creating a == To monitor the VMDK snapshot == To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command: # watch -d 'ls -luth | grep -E "delta|flat|sesparse"' # watch -d 'ls -luth | grep -E "delta|flat|sesparse"' == Change DNS server == {{{ ansible nsx -m raw -a 'esxcli network ip dns server remove --server=192.168.66.181' ansible nsx -m raw -a 'esxcli network ip dns server add --server=192.168.66.181' }}} == Drop/Add vmnic from vDS == Add or remove network cards (known as vmnics) to or from a vNetwork Distributed Switch (vDS) using these commands: esxcfg-vswitch -Q vmnic -V dvPort_ID_of_vmnic dvSwitch # unlink/remove a vDS uplink esxcfg-vswitch -P vmnic -V unused_dvPort_ID dvSwitch # add a vDS uplink -Q|--del-dvp-uplink=uplink Delete an uplink from a DVPort on a DVSwitch. Must specify DVPort Id. -V|--dvp=dvport Specify a DVPort Id for the operation. -P|--add-dvp-uplink=uplink Add an uplink to a DVPort on a DVSwitch. Must specify DVPort Id. {{{ vicfg-vswitch -Q vmnic0 --dvp 18 dvs-LAN-1 vicfg-vswitch -P vmnic1 --dvp 19 vs-LAN-0 }}} == Mount nfs from cli == {{{ [root@localhost:~] esxcfg-nas --add nfs --host=192.168.26.181 --share=/mnt/nfs Connecting to NAS volume: nfs nfs created and connected. [root@localhost:~] df -h Filesystem Size Used Available Use% Mounted on NFS 196.7G 50.1G 146.7G 25% /vmfs/volumes/nfs [root@localhost:~] esxcfg-nas -d 181-nfs NAS volume 181-nfs deleted. }}} == clean sensor == https://kb.vmware.com/s/article/74584 esxcli hardware ipmi sdr list | grep -i switch == find out ESXi host update date == {{{ for i in $(find . -iname "esxupdate.log" -o -iname "esxcfg-vmknic_-l.txt" -type f); do grep -E -H -a "20497097|10.10.21" $i | head -1 | awk '{ print $1,$2,$3,$4,$5}' ; done ./esx-localhost-2025-03-25--09.48-64755438/var/run/log/esxupdate.log:2024-09-23T04:09:07Z esxupdate: 21409970: imageprofile: INFO: ./esx-localhost-2025-03-25--09.48-64755438/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.97 ./esx-localhost-2025-03-25--10.12-54837774/var/run/log/esxupdate.log:2024-09-18T08:43:37Z esxupdate: 2101375: Transaction: DEBUG: ./esx-localhost-2025-03-25--10.12-54837774/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.78 ./esx-localhost-2025-03-25--10.14-74426764/var/run/log/esxupdate.log:2024-07-05T08:17:09Z esxupdate: 7872002: Transaction: DEBUG: ./esx-localhost-2025-03-25--10.14-74426764/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.61 ./esx-localhost-2025-03-25--09.58-57291123/var/run/log/esxupdate.log:2024-09-29T19:17:27Z esxupdate: 2101938: esxupdate: INFO: ./esx-localhost-2025-03-25--09.58-57291123/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.81 ./esx-localhost-2025-03-25--09.39-57646610/var/run/log/esxupdate.log:2024-09-12T05:46:23Z esxupdate: 2119344: imageprofile: INFO: ./esx-localhost-2025-03-25--09.39-57646610/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.74 ./esx-localhost-2025-03-25--10.05-72894411/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.67 ./esx-localhost-2025-03-25--09.51-73524741/var/run/log/esxupdate.log:2024-07-11T01:42:29Z esxupdate: 2101091: imageprofile: DEBUG: ./esx-localhost-2025-03-25--09.51-73524741/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.65 ./esx-localhost-2025-03-25--09.41-65846082/var/run/log/esxupdate.log:2024-09-23T02:47:30Z esxupdate: 2101544: imageprofile: DEBUG: ./esx-localhost-2025-03-25--09.41-65846082/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.96 ./esx-localhost-2025-03-25--10.19-80364452/var/run/log/esxupdate.log:2024-07-11T04:47:25Z esxupdate: 2101125: Transaction: DEBUG: ./esx-localhost-2025-03-25--10.19-80364452/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.68 ./esx-localhost-2025-03-25--09.45-84267988/var/run/log/esxupdate.log:2024-07-22T02:47:43Z esxupdate: 248268740: Transaction: DEBUG: ./esx-localhost-2025-03-25--09.45-84267988/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.69 ./esx-localhost-2025-03-25--10.16-79714476/var/run/log/esxupdate.log:2024-07-05T08:40:18Z esxupdate: 2101594: Transaction: DEBUG: ./esx-localhost-2025-03-25--10.16-79714476/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.64 }}} |
Contents
ESXi
Get all VM power state and more
- esxcli vm process list
for vmid in vim-cmd vmsvc/getallvms | awk '{ print $1 }' | grep -v Vmid; do echo $vmid; done | awk 1 ORS=' '
for vmid in `vim-cmd vmsvc/getallvms | awk '{ print $1 }' | grep -v Vmid`; do vim-cmd vmsvc/get.summary $vmid | egrep -i 'ipAddress|vim.VirtualMachine|powerState|
hostname =';done
Rescan storage
Initiate a storage rescan on the ESXi host through the vCenter Server GUI by right click on the ESXi host and select Storage > Rescan Storage) or through the command line by executing this command:
vmkfstools -V
identify DEVICE ERROR CODE
Beacon Probing Deep-Dive
With 3 or more uplinks in a team, we can pin point failures of a single uplink. With 2 uplinks in a team, we can detect downstream link failure, but we don't know which one is good and which bad.
- ESXi with two NICs
If you have three or more NICs, the mechanism is pretty straight-forward, but what happens if you have only two NICs?
In this situation, you cannot determinate if the problem is caused by the sending NIC ort he receiving NIC.
There is a fallback mechanism that will be used in this situation. Since ESXi does not know which of the NIC is affected (but it knows that one of the NIC is affected), it will simply starts to duplicate the frames on both NICs
iSCSI
esxcli iscsi adapter discovery sendtarget list
Grub boot ESXi6.5u2
- grub.cfg add ESXi6
set root=(hd0,gpt1) search --set=root --file /efi/VMware/safebt64.efi chainloader /efi/VMware/safebt64.efi
Build Customizer ISO
https://www.v-front.de/p/esxi-customizer-ps.html#download
ESXi-Customizer-PS-v2.6.0.ps1 -v65 -pkgDir c:\net-r816X -izip update-from-esxi6.5-6.5_update02.zip
Show Memory
smbiosDump | grep -A 12 -B 1 'Location: "DIMM' | egrep 'Location:|Bank:|Part Number:|Size:|Speed:|--'
ESXi 5.0
show arp
esxcli network ip neighbor list
show FRU
Get the source from sourceforge & unpack it
# curl "http://liquidtelecom.dl.sourceforge.net/project/ipmitool/ipmitool/1.8.15/ipmitool-1.8.15.tar.bz2" > ipmi.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 729k 100 729k 0 0 260k 0 0:00:02 0:00:02 --:--:-- 260k# bzip2 -dc ipmi.tar.gz | tar -xf -
compile
# cd ipmitool-1.8.15/ # ./configure CFLAGS=-m32 LDFLAGS=-static checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu . . . #make
Scp the file and run tool from ESX host
# scp src/ipmitool root@192.168.39.12:~
RUN IT
esxcfg-scsidevs -l | egrep -i 'display name|vendor|size' | head -20 /vmfs/volumes/view-2/ipmitool fru | head
References
To power on a virtual machine from the command line:
List the inventory ID of the virtual machine with the command:
vim-cmd vmsvc/getallvms |grep <vm name>
Note: The first column of the output shows the vmid.
Check the power state of the virtual machine with the command:
vim-cmd vmsvc/power.getstate <vmid>
Power-on the virtual machine with the command:
vim-cmd vmsvc/power.on <vmid> for vmid in `vim-cmd /vmsvc/getallvms | tail -n+2 | awk '{print $1}' | grep -o '[0-9]*'`; do vim-cmd vmsvc/power.getstate $vmid; done
Troubleshooting
ESXi network card driver
esxcli network nic get -n vmnic0
scratch warning
- Creating a
To monitor the VMDK snapshot
To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command:
# watch -d 'ls -luth | grep -E "delta|flat|sesparse"'
# watch -d 'ls -luth | grep -E "delta|flat|sesparse"'
Change DNS server
ansible nsx -m raw -a 'esxcli network ip dns server remove --server=192.168.66.181' ansible nsx -m raw -a 'esxcli network ip dns server add --server=192.168.66.181'
Drop/Add vmnic from vDS
Add or remove network cards (known as vmnics) to or from a vNetwork Distributed Switch (vDS) using these commands:
esxcfg-vswitch -Q vmnic -V dvPort_ID_of_vmnic dvSwitch # unlink/remove a vDS uplink
esxcfg-vswitch -P vmnic -V unused_dvPort_ID dvSwitch # add a vDS uplink
-Q|--del-dvp-uplink=uplink Delete an uplink from a DVPort on a DVSwitch.
- Must specify DVPort Id.
-V|--dvp=dvport Specify a DVPort Id for the operation.
-P|--add-dvp-uplink=uplink Add an uplink to a DVPort on a DVSwitch.
- Must specify DVPort Id.
vicfg-vswitch -Q vmnic0 --dvp 18 dvs-LAN-1 vicfg-vswitch -P vmnic1 --dvp 19 vs-LAN-0
Mount nfs from cli
[root@localhost:~] esxcfg-nas --add nfs --host=192.168.26.181 --share=/mnt/nfs Connecting to NAS volume: nfs nfs created and connected. [root@localhost:~] df -h Filesystem Size Used Available Use% Mounted on NFS 196.7G 50.1G 146.7G 25% /vmfs/volumes/nfs [root@localhost:~] esxcfg-nas -d 181-nfs NAS volume 181-nfs deleted.
clean sensor
https://kb.vmware.com/s/article/74584
esxcli hardware ipmi sdr list | grep -i switch
find out ESXi host update date
for i in $(find . -iname "esxupdate.log" -o -iname "esxcfg-vmknic_-l.txt" -type f); do grep -E -H -a "20497097|10.10.21" $i | head -1 | awk '{ print $1,$2,$3,$4,$5}' ; done
./esx-localhost-2025-03-25--09.48-64755438/var/run/log/esxupdate.log:2024-09-23T04:09:07Z esxupdate: 21409970: imageprofile: INFO:
./esx-localhost-2025-03-25--09.48-64755438/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.97
./esx-localhost-2025-03-25--10.12-54837774/var/run/log/esxupdate.log:2024-09-18T08:43:37Z esxupdate: 2101375: Transaction: DEBUG:
./esx-localhost-2025-03-25--10.12-54837774/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.78
./esx-localhost-2025-03-25--10.14-74426764/var/run/log/esxupdate.log:2024-07-05T08:17:09Z esxupdate: 7872002: Transaction: DEBUG:
./esx-localhost-2025-03-25--10.14-74426764/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.61
./esx-localhost-2025-03-25--09.58-57291123/var/run/log/esxupdate.log:2024-09-29T19:17:27Z esxupdate: 2101938: esxupdate: INFO:
./esx-localhost-2025-03-25--09.58-57291123/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.81
./esx-localhost-2025-03-25--09.39-57646610/var/run/log/esxupdate.log:2024-09-12T05:46:23Z esxupdate: 2119344: imageprofile: INFO:
./esx-localhost-2025-03-25--09.39-57646610/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.74
./esx-localhost-2025-03-25--10.05-72894411/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.67
./esx-localhost-2025-03-25--09.51-73524741/var/run/log/esxupdate.log:2024-07-11T01:42:29Z esxupdate: 2101091: imageprofile: DEBUG:
./esx-localhost-2025-03-25--09.51-73524741/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.65
./esx-localhost-2025-03-25--09.41-65846082/var/run/log/esxupdate.log:2024-09-23T02:47:30Z esxupdate: 2101544: imageprofile: DEBUG:
./esx-localhost-2025-03-25--09.41-65846082/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.96
./esx-localhost-2025-03-25--10.19-80364452/var/run/log/esxupdate.log:2024-07-11T04:47:25Z esxupdate: 2101125: Transaction: DEBUG:
./esx-localhost-2025-03-25--10.19-80364452/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.68
./esx-localhost-2025-03-25--09.45-84267988/var/run/log/esxupdate.log:2024-07-22T02:47:43Z esxupdate: 248268740: Transaction: DEBUG:
./esx-localhost-2025-03-25--09.45-84267988/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.69
./esx-localhost-2025-03-25--10.16-79714476/var/run/log/esxupdate.log:2024-07-05T08:40:18Z esxupdate: 2101594: Transaction: DEBUG:
./esx-localhost-2025-03-25--10.16-79714476/commands/esxcfg-vmknic_-l.txt:vmk0 Management Network IPv4 10.10.21.64