Contents
Optimization
http://www.yellow-bricks.com/2011/06/23/disk-schednumreqoutstanding-the-story/
Here's what we've tried:
- it is not the path: we suppressed the network switches (Nexus 5K) and plugged the storage directly into the ESXi server.
- We played disabling/enabling Delayed ACKs.
- We played disabling/enabling TSO/LRO
- We tuned the queue depth
- We tuned the ixgbe driver for the Intel 82599, disabling interrupt moderation
- We played with multipath: disabled it, re-enabled it, changes the IOPS rotation to 1.
- We changed the TCP/IP stack heap size to give it more room.
- Upon monitoring, we don't see the ESXi network interfaces as being busy, nor the iSCSI heads on the other end being busy. Low network, low CPU, and low Storage utilization on the SAN.
To identify the storage adapter queue depth
Run the esxtop command in the service console of the ESX host or the ESXi shell (Tech Support mode). For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910) or Tech Support Mode for Emergency Support (1003677) .
Press d.
Press f and select Queue Stats.
The value listed under AQLEN is the queue depth of the storage adapter. This is the maximum number of ESX VMKernel active commands that the adapter driver is configured to support.
To identify the storage device queue depth:
Run the esxtop command in the service console of the ESX host or the ESXi shell (Tech Support mode). For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910) or Tech Support Mode for Emergency Support (1003677).
Press u.
Press f and select Queue Stats.
The value listed under DQLEN is the queue depth of the storage device. This is the maximum number of ESX VMKernel active commands that the device is configured to support.
Notes:
The value listed under LQLEN is the LUN queue depth. This is the maximum number of ESX/ESXi VMkernel active commands supported by the LUN.
The value listed under %USD is the percentage of queue depth (adapter, LUN, or world) used by ESX/ESXi VMkernel active commands.
Changing the queue depth for QLogic, Emulex, and Brocade HBAs
http://www.vladan.fr/some-answers-about-queues-lun-and-hba-queue-depth-on-esxi-hosts/
- Changing the Queue Depth for QLogic and Emulex HBAs
“The recommendation is to set both Disk.SchedNumReqOutstanding and the adapter queue depth to the same value.”
https://kb.netapp.com/support/index?page=content&id=1013944
Verify which HBA module is currently loaded by entering one of these commands on the service console:
For QLogic:
# esxcli system module list | grep qla
For ESXi 5.5 QLogic native drivers:
# esxcli system module list | grep qln
For Emulex:
# esxcli system module list | grep lpfc
For Brocade:
# esxcli system module list | grep bfa
Run one of these commands:
Note: The examples show the QLogic qla2xxx and Emulex lpfc820 modules. Use the appropriate module based on the outcome of the previous step.
For QLogic:
# esxcli system module parameters set -p ql2xmaxqdepth=64 -m qla2xxx
For ESXi 5.5 QLogic native drivers:
# esxcli system module parameters set -p ql2xmaxqdepth=64 -m qlnativefc
For Emulex:
# esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc820
For ESXi 5.5 Emulex native drivers:
# esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc
For Brocade:
# esxcli system module parameters set -p bfa_lun_queue_depth=64 -m bfa
Notes:
In these commands, both ql2xmaxqdepth and lpfc0 use the lowercase letter L, "l", and not the numeric digit 1.
In this case, the HBAs represented by ql2x and lpfc0 have their LUN queue depths set to 64.
If all Emulex cards on the host must be updated, apply the global parameter, lpfc_lun_queue_depth instead.
Reboot your host.
Run this command to confirm that your changes have been applied:
# esxcli system module parameters list -m driver
Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.
The output appears similar to:
Name Type Value Description
-------------------------- ---- ----- --------------------------------------------------
.....
ql2xmaxqdepth int 64 Maximum queue depth to report for target devices.
How to increase the maximum queue depth per Logical Unit Number
https://kb.netapp.com/index?page=content&id=1011356&pmv=print&impressions=false
The queue depth parameter can also be changed using the SANsurfer utility.
Steps:
Perform the following steps:
- Run the SANsurfer HBA manager utility.
Click on HBA port > Settings. Click Advanced HBA port settings in the drop down list box. Update the Execution Throttle parameter.
To update Emulex HBA queue depths On Windows hosts, you can use the LPUTILNT utility to update the queue depth for Emulex HBAs or Registry edit.
Option 1:
- Run the LPUTILNT utility located in the c:\\WINNT\system32 directory. Select Drive Parameters from the pull down menu on the right side.
Scroll down and double-click QueueDepth.
VMware KB: Setting the Maximum Outstanding Disk Requests for virtual machines
Notes:
These procedures only apply to the ESXi/ESX host that the parameters are changed on. You must make the same changes to all other ESXi/ESX hosts that have the datastore/LUN presented to them.
- Ensure that the current value of any parameters are recorded before changing them in case you wish to revert back to their default values.
Warning: This change can have a major impact on the performance of your storage array. Use extreme caution and only adjust this parameter on the recommendation of your storage vendor.
To set the VMkernel limit for a particular LUN/device in vSphere 5.5 using the command line:
Open an SSH session to the host. For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910). To check the current value for a device,run the command:
- esxcli storage core device list -d naa.xxx
Note: The value appears under No of outstanding IOs with competing worlds:
To modify the current value for a device,run the command
- esxcli storage core device set -d naa.xxx -O Value
Where Value is between 1 and 256
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1268
http://www.boche.net/blog/index.php/2011/06/16/disk-schednumreqoutstanding-and-queue-depth
Enabling Storage I/O Control
To enable SIOC:
Select a datastore in the vSphere Client inventory and click the Configuration tab. Click Properties. Under Storage I/O Control, select Enabled. Click Close.
- Note: This setting is specific to the datastore and not to the host.
If you experience problems with SIOC or if the number of hosts connected to the datastore has changed since enabling SIOC:
Disable SIOC and save the changes by clicking OK.
Enable SIOC and save the changes. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022091
Cisco UCS QDLEN setting
http://virtuallyhyper.com/2012/07/seeing-lun-queue-depth-filling-up-with-ucs-b300s-and-emc-vmax/ === certified 5.5 I/O drivers are compatible with vSphere 6.0 ==
https://supportforums.cisco.com/discussion/11744776/ucs-hba-queue-depth-vmware-esxi-51
- FNIC driver download
Lists the hosts HBAs and the associated driver name
- esxcfg-scsidev -a
- vmkload_mod -s fnic
Run this command to view the current configuraiton:
- esxcli system module parameters list -m fnic
Name Type Value Description -------------------- ---- ----- ------------------------------------------------------------------------- fnic_max_qdepth uint Queue depth to report for each LUN fnic_trace_max_pages uint Total allocated memory pages for fnic trace buffer heap_initial int Initial heap size allocated for the driver. heap_max int Maximum attainable heap size for the driver. skb_mpool_initial int Driver's minimum private socket buffer memory pool size. skb_mpool_max int Maximum attainable private socket buffer memory pool size for the driver.
Run this command to change the queue depth to 64:
- esxcli system module parameters set -p fnic_max_qdepth=64 -m fnic
Run this command to check it
- esxcli system module parameters list -m fnic
Name Type Value Description -------------------- ---- ----- ------------------------------------------------------------------------- fnic_max_qdepth uint 64 Queue depth to report for each LUN fnic_trace_max_pages uint Total allocated memory pages for fnic trace buffer heap_initial int Initial heap size allocated for the driver. heap_max int Maximum attainable heap size for the driver. skb_mpool_initial int Driver's minimum private socket buffer memory pool size. skb_mpool_max int Maximum attainable private socket buffer memory pool size for the driver.
Cisco bug