VMware   Leave a comment

Resolving SCSI reservation conflicts

Details

  • You cannot access a LUN.
  • The LUN is zoned, presented and configured properly.
  • A vdf command never completes.

Solution

This article provides instructions for:

Note: For more information about why SCSI reservations occur, see Analyzing SCSI Reservation conflicts on VMware Infrastructure 3.x and vSphere 4.x (1005009).

ESX/ESXi 4.x and 5.0

To identify if SCSI reservation conflicts are preventing a LUN from being accessed:
  1. Verify that the LUN is detected by the ESX host at boot time. Run the command:# esxcfg-scsidevs -c

    The output appears similar to:

    Device UID Device Type Console Device Size Plugin Display Name
    eui.00000e1100b81bfc Direct-Access /dev/sda 70007MB NMP Local FUJITSU Disk (eui.00000e1100b81bfc)
    mpx.vmhba0:C0:T0:L0 CD-ROM /dev/sr0 0MB NMP Local HL-DT-ST CD-ROM (mpx.vmhba0:C0:T0:L0)
    mpx.vmhba3:C0:T0:L0 Direct-Access /dev/cciss/c0d0 34727MB NMP Local VMware Disk (mpx.vmhba3:C0:T0:L0)

  2. If the LUN is not listed, rescan the storage. For more information, see Performing a rescan of the storage (1003988).
  3. Check the system logs for signs of SCSI Reservation Conflict errors. Look for lines that may provide some information if the LUN is having an issue:ESX 4.x: # cat /var/log/dmesg

    ESXi 4.x: # cat /var/log/messages
    ESXi 5.0: # cat /var/log/vmkernel.log

    Vendor: EMC Model: SYMMETRIX Rev: 5771
    Type: Direct-Access ANSI SCSI revision: 03
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    sdh : READ CAPACITY failed.
    status = c, message = 00, host = 0, driver = 00
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because this path could not complete a READ
    command eventhough a TUR worked.
    result = 0x18 key = 0x0, asc = 0x0, ascq = 0x0
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because it is a duplicate path or on a passive
    path
    scan_scsis starting finish
    scan_scsis done with finish     

    In this example, LUN 52 is inaccessible in the ESX host cluster. Since it is listed in the output of step one, it was accessible at some point (a host reserved the LUN and never released it, possibly due to a SAN switch reboot in the middle of the reservation operation).

To resolve this:
  1. Examine all hosts (also applies to hosts for any pending reservations) with the command:# esxcfg-info | egrep -B5 “s Reserved|Pending”

    The output appears similar to:

    Note: Not all of the output is shown.

    —-Console Device…………………./dev/sdd
    |—-DevfsPath……………………..    /vmfs/devices/disks/vml.
    02000000006001c230d8abfe000ff76c51486715db504552432035
    |—-SCSI Level……………………..6
    |—-Queue Depth…………………….128
    |—-Is Pseudo………………………false
    |—-Is Reserved…………………….false
    |—-Pending Reservations…………….0
    —
    |—-Console Device…………………./dev/sda
    |—-DevfsPath……………………../vmfs/devices/disks/vml.
    02000000006001c230d8abfe000ff76c198ddbc13e504552432035
    |—-SCSI Level……………………..6
    |—-Queue Depth…………………….128
    |—-Is Pseudo………………………false
    |—-Is Reserved…………………….false
    |—-Pending Reservations…………….    1

    The ESX host that has Pending Reserves with a value that is larger than 0 is holding the lock.

  2. Perform a LUN reset to clear the lock with the command:# vmkfstools –lock lunreset /vmfs/devices/disks/vml.02000000006001c230d8abfe000ff76c198ddbc13e504552432035

  3. Verify that the LUN no longer has any Pending Reserves with the command:# esxcfg-info -s | egrep -B16 “s Reserved|Pending”

    Note: If the Pending Reserves is not 0 or Is Reserved is not false, the LUN reset did not succeed.

  4. Do a refresh of the storage window on each ESX host.
  5. If the datastore is not mounted, perform a rescan. For more information, see Performing a rescan of the storage (1003988).
Note: Virtual machines housed on a LUN should observe no interruption in service when a LUN reset occurs.

ESX/ESXi 3.5.x

To identify if SCSI reservation conflicts are preventing a LUN from being accessed:
  1. Verify that the LUN is detected by the ESX host at boot time with the commands:# cd /vmfs/devices/disks
    # ls vmh*

    The output appears similar to:

    vmhba0:0:0:0
    vmhba0:0:43:0
    vmhba0:0:52:0

     

  2. If the LUN is not listed, rescan the storage. For more information, see Performing a rescan of the storage (1003988).
  3. Checkdmesg.Look for lines that may provide some information if the LUN is having an issue:# cd /var/log
    # cat dmesg

    Vendor: EMC Model: SYMMETRIX Rev: 5771
    Type: Direct-Access ANSI SCSI revision: 03
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    sdh : READ CAPACITY failed.
    status = c, message = 00, host = 0, driver = 00
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because this path could not complete a READ
    command eventhough a TUR worked.
    result = 0x18 key = 0x0, asc = 0x0, ascq = 0x0
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because it is a duplicate path or on a passive
    path
    scan_scsis starting finish
    scan_scsis done with finish

    In this example, LUN 52 is inaccessible in the ESX host cluster. Since it is listed in the output of step one, it was accessible at some point (a host reserved the LUN and never released it, possibly due to a SAN switch reboot in the middle of the reservation operation).

To resolve this:
  1. Examine all hosts (also applies to hosts for any pending reservations) with the command:# esxcfg-info | egrep -B5 “s Reserved|Pending”

    The output appears similar to:

    Note: Not all of the output is shown.

    |—-Console Device…………………./dev/sdd
    |—-DevfsPath……………………..    /vmfs/devices/disks/vml.
    02000000006001c230d8abfe000ff76c51486715db504552432035
    |—-SCSI Level……………………..6
    |—-Queue Depth…………………….128
    |—-Is Pseudo………………………false
    |—-Is Reserved…………………….false
    |—-Pending Reservations…………….0
    —
    |—-Console Device…………………./dev/sda
    |—-DevfsPath……………………../vmfs/devices/disks/vml.
    02000000006001c230d8abfe000ff76c198ddbc13e504552432035
    |—-SCSI Level……………………..6
    |—-Queue Depth…………………….128
    |—-Is Pseudo………………………false
    |—-Is Reserved…………………….false
    |—-Pending Reservations…………….    1

    The ESX host that has Pending Reserves with a value that is larger than 0 is holding the lock.

  2. Perform a LUN reset to clear the lock with the command:# vmkfstools –lock lunreset /vmfs/devices/disks/vml.02000000006001c230d8abfe000ff76c198ddbc13e504552432035

  3. Verify that the LUN no longer has any Pending Reserves with the command:# esxcfg-info -s | egrep -B5 “s Reserved|Pending”

    Note: If the Pending Reserves is not 0 or Is Reserved is not false, the LUN reset did not succeed.

  4. Do a refresh of the storage window on each ESX host.
  5. If the datastore is not mounted, perform a rescan. For more information, see Performing a rescan of the storage (1003988).

ESX 3.0.x

To identify if SCSI reservation conflicts are preventing a LUN from being accessed:
  1. Verify that the LUN is detected by the ESX host at boot time. Run the commands:# cd /vmfs/devices/disks
    # ls vmh*

    The output appears similar to:

    vmhba0:0:0:0
    vmhba0:0:43:0
    vmhba0:0:52:0

     

  2. If the LUN is not listed, rescan the storage. For more information, see Performing a rescan of the storage (1003988).
  3. Checkdmesg .Look for lines that may provide some information if the LUN is having an issue:# cd /var/log
    # cat dmesg

    Vendor: EMC Model: SYMMETRIX Rev: 5771
    Type: Direct-Access ANSI SCSI revision: 03
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    sdh : READ CAPACITY failed.
    status = c, message = 00, host = 0, driver = 00
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    scsi0 (0,0,52) : RESERVATION CONFLICT
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because this path could not complete a READ
    command eventhough a TUR worked.
    result = 0x18 key = 0x0, asc = 0x0, ascq = 0x0
    VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
    channel 0, id 0, lun 52
    Has not been attached because it is a duplicate path or on a passive
    path
    scan_scsis starting finish
    scan_scsis done with finish

    In this example, LUN 52 is inaccessible in the ESX host cluster. Since it is listed in the output of step one, it was accessible at some point (a host reserved the LUN and never released it, possibly due to a SAN switch reboot in the middle of the reservation operation).

To resolve this:
  1. Examine all hosts for any pending reservations with the command:# tail -1 /proc/vmware/scsi/vmhba[0-9]/[0-9]:*

    The output appears similar to:

    ==> /proc/vmware/scsi/vmhba0/0:0 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba0/0:43 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba0/0:44 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba0/0:45 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba0/0:46 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba1/0:50 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba1/0:51 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba1/0:52 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 1

    ==> /proc/vmware/scsi/vmhba1/0:53 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba1/0:54 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    ==> /proc/vmware/scsi/vmhba2/0:0 <==
    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    The ESX host that has Pending Reserves with a value that is larger than 0 is holding the lock.

  2. Perform a LUN reset to clear the lock with the command:# vmkfstools –lock lunreset /vmfs/devices/disks/vmhba1\:0\:52\:0

  3. Verify that the LUN no longer has any Pending Reserves with the command:# tail -1 /proc/vmware/scsi/vmhba1/0\:52

    The output appears similar to:

    Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

    Note: If the Pending Reserves is not 0 or Is Reserved is not false, the LUN reset did not succeed.

  4. Do a refresh of the storage window on each ESX host.
  5. If the datastore is not mounted, perform a rescan. For more information, see Performing a rescan of the storage (1003988).

Posted May 11, 2012 by g6237118

Leave a comment