http://blogs.vmware.com/vsphere/2012/07/troubleshooting-storage-performance-in-vsphere-part-5-storage-queues.html

Optimization

http://www.yellow-bricks.com/2011/06/23/disk-schednumreqoutstanding-the-story/

Here's what we've tried:

- it is not the path: we suppressed the network switches (Nexus 5K) and plugged the storage directly into the ESXi server.

- We played disabling/enabling Delayed ACKs.

- We played disabling/enabling TSO/LRO

- We tuned the queue depth

- We tuned the ixgbe driver for the Intel 82599, disabling interrupt moderation

- We played with multipath: disabled it, re-enabled it, changes the IOPS rotation to 1.

- We changed the TCP/IP stack heap size to give it more room.

- Upon monitoring, we don't see the ESXi network interfaces as being busy, nor the iSCSI heads on the other end being busy. Low network, low CPU, and low Storage utilization on the SAN.

To identify the storage adapter queue depth

    Run the esxtop command in the service console of the ESX host or the ESXi shell (Tech Support mode). For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910) or Tech Support Mode for Emergency Support (1003677) .
    Press d.
    Press f and select Queue Stats.
    The value listed under AQLEN is the queue depth of the storage adapter. This is the maximum number of ESX VMKernel active commands that the adapter driver is configured to support. 

To identify the storage device queue depth:

    Run the esxtop command in the service console of the ESX host or the ESXi shell (Tech Support mode). For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910) or Tech Support Mode for Emergency Support (1003677).
    Press u.
    Press f and select Queue Stats.
    The value listed under DQLEN is the queue depth of the storage device. This is the maximum number of ESX VMKernel active commands that the device is configured to support. 

Notes:

    The value listed under LQLEN is the LUN queue depth. This is the maximum number of ESX/ESXi VMkernel active commands supported by the LUN.
    The value listed under %USD is the percentage of queue depth (adapter, LUN, or world) used by ESX/ESXi VMkernel active commands. 

Changing the queue depth for QLogic, Emulex, and Brocade HBAs

    Verify which HBA module is currently loaded by entering one of these commands on the service console:

        For QLogic:

        # esxcli system module list | grep qla

        For ESXi 5.5 QLogic native drivers:

        # esxcli system module list | grep qln

        For Emulex:

        # esxcli system module list | grep lpfc

        For Brocade:

        # esxcli system module list | grep bfa

    Run one of these commands:

    Note: The examples show the QLogic qla2xxx and Emulex lpfc820 modules. Use the appropriate module based on the outcome of the previous step.

        For QLogic:

        # esxcli system module parameters set -p ql2xmaxqdepth=64 -m qla2xxx

        For ESXi 5.5 QLogic native drivers:

        # esxcli system module parameters set -p ql2xmaxqdepth=64 -m qlnativefc

        For Emulex:

        # esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc820

        For ESXi 5.5 Emulex native drivers:

        # esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc

        For Brocade:

        # esxcli system module parameters set -p bfa_lun_queue_depth=64 -m bfa

    Notes:
        In these commands, both ql2xmaxqdepth and lpfc0 use the lowercase letter L, "l", and not the numeric digit 1.
        In this case, the HBAs represented by ql2x and lpfc0 have their LUN queue depths set to 64.
        If all Emulex cards on the host must be updated, apply the global parameter, lpfc_lun_queue_depth instead.

    Reboot your host.
    Run this command to confirm that your changes have been applied:

    # esxcli system module parameters list -m driver

    Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.

    The output appears similar to:

    Name                        Type  Value  Description 
    --------------------------  ----  -----  --------------------------------------------------
    .....
    ql2xmaxqdepth               int   64     Maximum queue depth to report for target devices.

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1267

How to increase the maximum queue depth per Logical Unit Number

https://kb.netapp.com/index?page=content&id=1011356&pmv=print&impressions=false

The queue depth parameter can also be changed using the SANsurfer utility.

Steps:

Perform the following steps:

• Run the SANsurfer HBA manager utility.

  Click on HBA port > Settings. Click Advanced HBA port settings in the drop down list box. Update the Execution Throttle parameter.

To update Emulex HBA queue depths On Windows hosts, you can use the LPUTILNT utility to update the queue depth for Emulex HBAs or Registry edit.

Option 1:

• Run the LPUTILNT utility located in the c:\\WINNT\system32 directory. Select Drive Parameters from the pull down menu on the right side.

  Scroll down and double-click QueueDepth.

VMware KB: Setting the Maximum Outstanding Disk Requests for virtual machines

Notes:

These procedures only apply to the ESXi/ESX host that the parameters are changed on. You must make the same changes to all other ESXi/ESX hosts that have the datastore/LUN presented to them.

• Ensure that the current value of any parameters are recorded before changing them in case you wish to revert back to their default values.

Warning: This change can have a major impact on the performance of your storage array. Use extreme caution and only adjust this parameter on the recommendation of your storage vendor.

To set the VMkernel limit for a particular LUN/device in vSphere 5.5 using the command line:

Open an SSH session to the host. For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910). To check the current value for a device,run the command:

• esxcli storage core device list -d naa.xxx

Note: The value appears under No of outstanding IOs with competing worlds:

To modify the current value for a device,run the command

• esxcli storage core device set -d naa.xxx -O Value

Where Value is between 1 and 256

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1268

http://www.boche.net/blog/index.php/2011/06/16/disk-schednumreqoutstanding-and-queue-depth