VMAX_”Gatekeepers”

1. What are the most important things to know about Gatekeepers?

• If the system selects and uses a data or system device as a Gatekeeper, applications on the application host can be impacted! EMC STRONGLY recommends that dedicated Gatekeepers be created and used. This is particularly important with Open Systems operation. See below.
• Gatekeeper operation for Open Systems and Mainframe operation is similar in principle but there are major differences. Read below for more information about Open Systems operation and refer to EMC Knowledgebase solution emc195898 for more information about Mainframe Gatekeeper operation.
• PowerPath can be used with Gatekeepers and there is a great reason to do so (refer to item 15 below). No other multi-pathing products can be used to define multiple paths to those Gatekeeper devices.

2. What is a Gatekeeper?

Solutions Enabler and Mainframe Enabler are EMC software components used to control the features of Symmetrix arrays. They receive user requests from CLI, GUI, or other means, and generate low-level commands (syscalls) that are transmitted into the Symmetrix array for action.

Gatekeeper devices are Symmetrix devices that act as the target of command requests to Enginuity based functionality. These commands arrive in the form of disk I/O requests (and so a “disk” must be named by the host as the “address” or target of that command). The more commands that are issued from the host, and the more complex the actions required by those commands are, the more gatekeepers and/or array processor resources that are required to handle those requests in a timely manner.

3. Do I really need to use Gatekeepers?

If Symmetrix commands are running successfully requesting a Symmetrix command, it means that a device is being used as a Gatekeeper; the command has to be sent to something the Host Bus Adaptor can address, that is, a device. The larger question is what is the nature of that Gatekeeper? Is it a user device used to hold data or is it a system paging disk? If so, there is a serious risk of application impact. This is why EMC strongly recommends the use of dedicated Gatekeepers, as described below.

4. Do I need to create Gatekeepers?

Open Systems:

Open Systems servers should use dedicated Symmetrix devices (that is, dedicated to being a Gatekeeper with no other purpose) for Gatekeeper purposes. Solutions Enabler selects a Gatekeeper device for a given command based upon a hierarchy of choices (found in Solutions Enabler documentation) and could be forced to utilize a device that is used by a database or by the operating system if not enough dedicated Gatekeepers are defined. If this happens, it can significantly impact applications that may be operating on a server.

A Gatekeeper device may not respond to an application or server I/O request until a Symmetrix command request is completed. If a Symmetrix is executing a complex or time-consuming command, such as a query of the Remote R2 Symmetrix, line latency may cause a delay and the device can, in effect, be locked for some time. If the time is too long, the application using that Gatekeeper as a data device may experience I/O timeouts or perform poorly.

Mainframe:

Mainframe generally requires users to name a device that is used as a target of a syscall. Some mainframe commands and applications can select devices from a pool. Refer to EMC Knowledgebase solution emc195898 for more information. The mainframe information in this Knowledgebase solution is only to illustrate differences between mainframe and open systems and is not complete.

Gatekeepers should be created if there are no existing devices already dedicated to Gatekeeper use. Refer to EMC Knowledgebase solution emc287007 for “How to use Solutions Enabler to create Mainframe Gatekeeper devices”.

5. Why does the Symmetrix use Gatekeepers and not some other means of control?

The Symmetrix is a high performance storage system that scales up to the largest storage capacities available in the industry. It also has very comprehensive monitoring, performance and diagnostic capabilities. As a result, command dialogs with the system can be extensive and can involve a good deal of data. Symmetrix uses Gatekeepers as a scalable, high-performance means of command and control, suitable to the capabilities of the array. The deterministic nature of disk I/O is particularly powerful when used for array-wide or multi-array operations such as consistent splits, in which conventional networking may encounter packet collisions or retries, which could prevent the timely operation of disaster recovery commands.

6. Are there devices that should NOT be used as Gatekeepers?
How do I do that?

Open Systems:

There are devices that should not be used as Gatekeepers. The rule of thumb is,”do not use devices that have user or system data on them” to avoid conflict and I/O performance issues when operating Symmetrix features. This is why EMC makes the strong recommendation to dedicate devices exclusively for Gatekeeper use. There are three means to explicitly control what devices will or will not be used as Gatekeepers:

• Devices that absolutely should not be used as Gatekeepers can be put into a gkavoid file as part of the Solutions Enabler setup.  This prevents devices in that file from being used as Gatekeepers, yet still permits Solutions Enabler the freedom to select other devices as Gatekeepers as required. Note that adding new devices to a server or application may require updating this file with those devices. Refer to EMC Knowledgebase solution emc11514 for Setting up gkavoid files.
• Devices intended for Gatekeeper use only can be put into the gkselect file. Devices in this file will take priority in the selection process. Solutions Enabler may choose data devices as a Gatekeeper for a particular Symmetrix if none its Gatekeepers in the gkselect file exist, or are offline.
• After determining how many gatekeepers are needed, create dedicated Gatekeepers, and set the storapid daemon  option, gk_use in the daemon_options file to specify dedicated only. (storapid:gk_use=dedicated_only)
• Meta Devices cannot be configured as Gatekeepers.

7. How many Gatekeepers do I need?

Open Systems:

The answer is, “Enough to perform all concurrent Symmetrix commands and queries without running into a condition in which commands are delayed or rejected for lack of a gatekeeper resource.” Practically speaking, any server issuing commands to a Symmetrix, physical or virtual, should have six Gatekeeper devices uniquely available to it. Details of their creation and characteristics are found elsewhere in this Knowledgebase solution.

Six Gatekeepers should prove sufficient for any Open Systems control server (meaning a server that is sending control syscalls to the Symmetrix instead of, or in addition to, data), except those running Consistency Groups (more in the paragraph below). More than six can be needed under very heavy management workloads, such as heavy use of ControlCenter Performance Manager and/or Symmetrix Performance Manager, or for certain other circumstances (refer to item 16 below).

For those locations utilizing Consistency Groups, add the following device count to the six devices mentioned above. Here is how to calculate the required Gatekeeper count:

• Solutions Enabler 7.0:
  In addition to the six above, another fourteen Gatekeepers must be added if there is performance-critical SRDF operation, plus for [Number of CGs] * 0.5 = Gatekeepers needed for Consistency Group operation.
• Solutions Enabler 7.1:
  [Number of CGs] * 0.5 = Gatekeepers needed for Consistency Group operation
• Solutions Enabler 7.2:
  [Number of CGs] * 0.3 = Gatekeepers needed for Consistency Group operation

Round up the result of calculations in bullets 2 and 3 above to whole integers and then do the following addition:
6 + (those needed for 7.0 with SRDF) + (GKs needed for CG operation) = total number of Gatekeepers to specify.

8. How big should Gatekeepers be?

Gatekeepers require only a small amount of space, 3 MB (3 cyl) for Enginuity levels 57xx, 58xx and higher, and 3 MB (6 cyl) for Enginuity levels of 56xx and lower. Users are encouraged to not build Gatekeepers in larger sizes as the small size is used by Enginuity to automatically identify and use devices as Gatekeepers. Devices under 8 MB have the indication GK in syminq output. Gatekeeper devices must be mapped and masked to single servers only and should not be shared across servers. For Virtual Server and Cluster environments, refer to the important additional information in item 16 below.

Gatekeepers should be RAID protected in some way so if the device were to fail, the syscall would be directed to the “mirror” of the failed device.  Any RAID format will provide this protection, but a simple mirror (RAID 1) is suggested.  Other RAID configurations take a bit more space, for example, to make it 7+1 involves spreading the gatekeeper across 8 drives (no actual I/O goes to the disk, so there is no ‘I/O’ penalty).  In such a case, your 3 cyl gatekeeper will take (7+1)*3= 24 cylinders.  A nit, but for completeness’ sake…

9. Can I share Gatekeepers across servers?

Gatekeeper devices must be mapped and masked to single hosts only and should not be shared for concurrent I/O across hosts. For Virtual Server and Cluster environments, refer to the important additional information in item 16 below.

10. Can I share server resources like HBAs between production applications and Gatekeepers?

Many configurations have Solutions Enabler and control commands running from a production control server. Just as production applications have requirements for minimal response times, certain Symmetrix management and control operations should be deemed real time and critical (consistency protection is a good example).

While gatekeepers can share HBAs and FA ports with production hosts, there are good reasons to isolate them to separate HBAs and FA ports.

• Perhaps most critical, consistency related commands needed for proper disaster protection may be held up by pending I/O’s previously issued by a shared HBA (some HBAs will stop sending until some of the pending I/Os finish).
• Applications on that HBA could be impacted by syscall traffic. An overworked server can also cause Symmetrix control issues, because the server itself can be too busy to send a time-critical control operation to the Symmetrix.

It is always a best practice to isolate Gatekeepers on production servers to HBA and FA ports that are not response time sensitive and are not busy with production I/O. It is possible for gatekeepers to share FA ports as long as a given Gatekeeper is used only by a given control server.

11. Does the number of needed Gatekeepers change for different software releases?

Open Systems:

The minimum number of Gatekeepers needed does change over time as EMC works to make control software more efficient. The numbers provided in this Knowledgebase solution can be used with available releases.

12. How do Open Systems and Mainframe Gatekeepers differ?

While the underlying principles involved are identical, the differing nature of Mainframe I/O and SCSI I/O results in behaviors that are not identical. Refer to EMC Knowledgebase solution emc195898 for Mainframe Gatekeeper information. This Knowledgebase solution is written with a focus on Open Systems and contains only partial Mainframe information.

13. Besides creating Gatekeepers, are there other things I need to set up to use them correctly?

Yes. Make them a suitable size (see above). Be sure they are not SRDF, or virtual or snap devices, or devices in a save pool or thin pool.

14. Can I use PowerPath with Gatekeepers and is there a reason to do so?

Yes, and there is good reason to do so. Many Open Systems customers double the Gatekeeper count, with half on each of two paths. With this configuration, in the event of path failure, the Symmetrix could still receive commands. PowerPath protects against path failure without the need to double the device count.

15. Can I use other multi-pathing products with Gatekeepers?

No. Other multi-pathing products may not be used to define multiple paths with Gatekeepers. PowerPath uniquely recognizes Symmetrix commands and treats them in a special manner when multiple paths are available to the Gatekeeper device. Products without this awareness can, for example, retry the I/O and cause significant issues with the command processing.

16. Virtualization/Cluster/Migration or Application Specific Gatekeeper Requirements.

• Virtual Server Environments
  Virtual Server environments often permit the movement of a guest operating system instance from one physical server to another. If that instance is doing Symmetrix management or control, the Gatekeepers being used must be visible to that instance on whatever physical server it is running upon. As a result, the Gatekeeper devices must be made visible to the various physical servers the guest may operate upon. Note that the rule for not sharing Gatekeepers remains, and that the guest operating system image with its Solutions Enabler instance must run on only one physical machine at any given time, and only one instance using the Gatekeeper at any one time. Please also see EMC Knowledgebase solution emc248427.
• Clustered Environments
  Clustered environments also involve the movement of Symmetrix management and control from one server to another as part of a failover process. There are different types of clusters and the reader should look at the ELab Host Connectivity Guide of the particular operating system for details of their cluster configuration. With shared nothing clusters, the Gatekeepers should be visible to all the physical servers that may issue commands to the Symmetrix, in order to continue operations after a failover from the controlling node.  And as above, commands to a given Gatekeeper must be issued by only one control server instance at a time. In a share everything cluster (OpenVMS for example), a Gatekeeper must me mapped and masked to only one, and only one, server, in order to restrict Gatekeeper usage to the one controlling server.
• SRDF/CE
  Each Symmetrix must be configured with the recommended number of gatekeepers per node, per SRDF side for SRDF/CE exclusive use, and must be configured per the above information. They should also have Windows disk signatures so that they are fully visible to the cluster. For performance reasons, it is recommended that an additional gatekeeper be associated with every SRDF/CE device group.
• PowerPath Migration Enabler (PPME)
  Gatekeeper devices should not be migrated. Gatekeepers should not be migrated by any means because the new location will have a different context than the old location, and commands issued to the migrated Gatekeeper could result in unintended consequences in the new array.
• ControlCenter Agent and Symmetrix Management Console (SMC)
  Both be serviced by the recommended Gatekeeper count above. Note that large specific workloads, such as often collecting performance data for Symmetrix Performance Analyzer or Performance Manager, could require more Gatekeepers for proper operation.