PowerHA and External Storage

This blog is written by Jenny Dervin, IBM PowerHA SystemMirror architect. See her in this new video from IBM.

 

The types of storage used by IBM i customers have become as varied as the types of business that run on IBM i. As virtualization becomes more common and clients move into an external storage environment, PowerHA has technologies that integrate the replication capabilities on those external storage servers with IBM i.

PowerHA SystemMirror for i is based on shared storage clustering technology featuring storage-level replication to support geographic dispersion. That means that replication takes place when data is written to disk. PowerHA determines which volumes on the storage to replicate by its location or address. Production data is placed in an independent auxiliary storage pool (IASP), similar to a volume group on AIX or a D drive on a PC. Whenever a sector is changed within that IASP, it is replicated to the target or backup disk. Object types outside of the IASP are be synchronized via PowerHA using Administrative Domain in order to provide a complete production-ready environment on the backup system.

Most external storage servers support some sort of replication technology. PowerHA supports the replication technology of DS8000, SAN Volume Controller (SVC) and Storwize V7000. When the IASP is located on one of those storage servers, the high availability environment is defined and controlled via IBM i interfaces, but the replication itself is offloaded to the storage server. The advantage of this type of replication is its simplicity and efficiency. No verification is needed, as the data is kept synchronized by the storage servers.

Each of these storage servers support synchronous replication, called Metro Mirror, as well as asynchronous replication, called Global Mirror. With Metro Mirror, the production and target copies of the IASP are guaranteed to be identical. There are distance limitations with Metro Mirror in order to achieve good production response time, depending on available bandwidth and disk write rates. Global Mirror has no distance limitation. With Global Mirror, there will be a slight lag between the production and target copies, but the target data is guaranteed to be consistent, and most customers are able to tune the delay down to only a few seconds. Many customers implement another PowerHA technology, called lun-level switching, to provide local high availability, and then use Global Mirror as a disaster recovery solution. Lun-level switching involves switching an IASP on external storage between multiple partitions or systems in order to provide protection from planned or unplanned outages of the IBM i system.

One more very useful external storage technology, also integrated in PowerHA, is FlashCopy. FlashCopy gives you the ability to create a very fast copy of an IASP which can then be varied on to another partition and used for backups, development, testing, etc. Most of our PowerHA customers with external storage take advantage of this technology to do their backups offline from their production system. I will discuss this technology in more detail in a future blog.

For more information on any of these technologies, please reference the PowerHA SystemMirror for IBM i Cookbook.

 

 

 

 

i Can … Encrypt my Backups with BRMS

What is software encryption with BRMS? 

Simply said, it’s a way to encrypt your data to tape, via Backup, Recovery and Media Services (BRMS) without special hardware! A great option for those who don’t want to purchase the hardware option (tape drives that do the encryption) and only have small amounts of data to encrypt. Users who have large amounts of data that need to be encrypted should look at the hardware option for encryption.

How do you get this feature?

You must purchase the following products:

• BRMS Advanced Feature (5761-BR1/5770-BR1 Option 2) 
  *This assumes that the base BRMS product is already installed. If not, then you’ll also need 5761-BR1/5770-BR1 Base and the MSE product 5761/5770-SS1 Option 18 Media and Storage Extensions.
• Cryptographic Service Provider (5761-SS1/5770-SS1 Option 44)

Got the products, now where do I start?

Once you have these products installed, then you have to set up your keys via the key management tools. These tools are available through commands or System i Navigator and the IBM Systems Director Web environment. Using these tools, you’ll need to set up and create a keystore file named Q1AKEYFILE in the BRMS user library QUSRBRM. Ensure that the file name is exactly as stated here. It’s case sensitive, so use uppercase. This will be the only file that the BRMS function will look for the encryption key. And since it resides in the BRMS user library, it will get backed up when doing a full QUSRBRM save, which should be a part of your BRMS backup strategy. 

Important!

Understanding the Cryptographic key management on IBM i is extremely important. Any mismanagement of these keys could result in lost data, since the data can only be restored with these keys. If the keys were lost or corrupted, then the data on tape would be useless. Refer to Cryptographic services key management to clearly understand the importance of these master keys as well as the required steps to ensure your data is truly encrypted and recoverable. 

To locate the topic in the IBM i 6.1 and 7.1 Information Center, expand IBM i 7.1 Information Center > Security > Cryptography .

Keystore file has been created. What’s next?

Once the keystore file has been created, then the BRMS media policy that will be used for backup needs to be set for encryption. In a BRMS media policy there are four parameters; Encrypt data, Keystore file, Keystore library, and Key record label. Simply specify *YES for Encrypt data, and fill in the keystore file information. Remember, currently BRMS only accepts one keystore file (QUSRBRM/Q1AKEYFILE) for backup and that will already be filled in for you. Trying to change it to another keystore file will cause an error. The Key record label is the one parameter you must fill in. The Key record label parameter specifies a unique identifier of a key record in a keystore file. You should have created this label when you created the keystore file. 

Below is an example of the parameters you will see in the media policy:

Once you have a media policy that supports encryption, you can specify it on any of the BRMS save commands. This will then encrypt the items being saved. The SAVSAVFBRM and DUPMEDBRM also support this type of media policy. You will have the capability to encrypt save file data onto media, just by specifying a media policy that supports encryption. 

Also, if you would rather not encrypt your data during your backup, but would like to encrypt the data when you duplicate it to other media, simply specify a media policy that supports encryption on the media policy parameter on the DUPMEDBRM command. 

Additional customization in the control group

In addition to all the ways you may encrypt data, further controls have also been provided in the backup and archive control groups that allow you to enable and disable encryption for each save item. This control has been made on the Edit option for control groups. When editing your control group, there is a new Advanced backup controls display, that gives you an Encrypt column, to encrypt or not encrypt a specific entry within the control group. 

The following is an example of this new control:

What happens on the recovery for these encrypted saves?

The BRMS database is updated with information on encrypted saves, so the knowledge is there when a key is needed to decrypt. BRMS also relies on the user to specify what keystore file to use for decrypting on the restore. The user can rely on the keystore file that was used on the save or specify the keystore file name in the BRMS Recovery policy. Even though the original backup would have used the standard BRMS QUSRBRM/Q1AKEYFILE, there may be instances where the BRMS keystore file had to be migrated or changed, and will have a new file name. Because of those instances, BRMS wanted to ensure the user had the capability to restore the data from renamed keystore files. 

Below is an example of the Recovery policy encryption parameters:

BRMS also shows the backed up items as encrypted in the WRKMEDIBRM display and also on the BRMS Recovery report.

Example screen shot of the WRKMEDIBRM display that shows encrypted backed up libraries:

 

 

BRMS Recovery report and Volume Summary report, shows what items are encrypted and what volumes on the recovery report have encrypted data. 

Notice that the Key Record Label can differ on one volume and the report lists them.

Are there any restrictions with this function?

Yes, there are restrictions in what data you can encrypt and what device you can encrypt to. They include: 

1. *IBM, *SAVSYS, *SAVSECDTA, *SAVCFG and any libraries beginning with the letter Q are not allowed to be encrypted in BRMS.

2. Be aware of a possible performance impact when encrypting data.

    ***Performance metrics are available on the BRMS webpage

3. BRMS does not support encryption on optical or virtual optical devices.

That’s how to do software encryption with BRMS!

This is a great solution for those customers with small amounts of data that need to be encrypted since it requires no special drive, and can be used to not only physical tape drives but also virtual tape drives. For more step-by-step information on setting BRMS media polices and control groups, refer to the Software Encryption section of Chapter 7: Tailoring your backup chapter in the Backup, Recovery, and Media Services for i publication (SC41-5345-07). Also check out our BRMS website for updated information.

Kristi Harney wrote this blog article. Kristi is a software developer for the IBM i strategic backup product, BRMS. She has been working on the BRMS development team for 12 years and has helped educate many new and existing users of BRMS on product features and usage. Thanks, Kristi!

i Can … Remove Disk Units Without an IPL at 7.1

It's tough to make the new release highlights list. A valuable capability that will help reduce planned outages that didn't make the cut for IBM i 7.1 announcement highlights is the capability to remove disk units from the partition without requiring an IPL or going into restricted state. While it’s documented in the 7.1 Information Center, you most likely wouldn’t find it if you didn't know to go look for it. 

Starting with 7.1, you have capability to concurrently remove disk units in the system ASP, user ASPs,   and on various independent ASPs. Prior to 7.1, you had to manual IPL to DST in order to remove disk units in the system ASP or a user ASP.  If you wanted to remove units from an independent ASP, the independent ASP had to be varied off.

Just think of the possibilities! If you want to upgrade your disk technology to the latest and greatest, you can now do it with only a single outage to migrate the load source unit for a single ASP system.  For user ASPs or independent ASPs, you can achieve a complete upgrade of all disk units with no outage at all. 

The capability is access via System Service Tools (SST).  You'll notice a change on the Work with Disk Configuration panel. Click the following link to view panel: Download ICan Work with Disk Config panel 6.1.10

After selecting option 12, you will see a new panel. Click the following link to view panel: Download ICan Work with Removing Units From Configuration 6.1.10

As you can see from the options presented, not only can you start the remove process but you can pause the remove and resume one that’s paused. You can cancel the remove either with or without a balance.  You can also use either the option provided on the service tools panel or WRKDSKSTS to see the status of the remove operation.

Planning before beginning a remove of units will assure success with low business impact. Most important is to ensure enough free space on units that will remain to absorb the GBs from the disks you are removing. If the ASP threshold is exceeded while the remove process is running, the remove will be paused. 

Other items to consider while planning:
•    IBM i 7.1 is required.
•    Any server or blade hardware that supports IBM i 7.1 can be utilized.
•    If a remove is in progress when the system is powered down, ended abnormally, or an IASP is varied off, the remove will be paused. The remove must be restarted by choosing the resume option.
•    The load source unit cannot be removed from the system ASP.
•    Not all drives can be removed from an ASP. At least one must remain.
•    Moving drives between ASPs is not supported.The units must be removed from one ASP then added to the other ASP.
•    All units selected for removal must belong to a single ASP.
•    Only one ASP can be operated on at a time. Other ASPs removes may be paused but only one can be running.
•    No changes have been made to the remove processing that occurs during a manual IPL to DST.  A remove started at DST must complete before the IPL can continue.

Additional information about the remove process and other enhancements that increase your system's availability can be found online.

I'd like to thank Sue Baker for writing this blog article. Sue is a Certified Consulting IT Specialist working for IBM's Americas Advanced Technical Skills organization. Sue is currently focused on Power Systems hardware performance in the arenas of processor, memory, disk subsystems, and I/O. Sue consults with customers in the challenging arena of managing the operating environment to provide highly available, highly performing, and highly resilient Power Systems infrastructure.

i Can …. Manage My Passwords with IBM PowerHA SystemMirror for i

This blog article was written by Lilo Bucknell. Lilo is a member of the IBM i Power HA design and development team.Thanks, Lilo!

Today I’m going to show you how to keep your passwords synchronized across multiple systems using some simple commands provided with IBM PowerHA SystemMirror for i. The IBM PowerHA SystemMirror for i LP was introduced in release 6.1 as the IBM i disk-based high availability and disaster-recovery solution. Part of that solution includes the application known as cluster administrative domain. Even if you aren’t ready for a full-blown HA/DR configuration, you can use cluster administrative domain as a simplified management tool for managing resources across multiple systems, including your user profiles and passwords. 

PoweHA Cluster Administrative Domain
The PowerHA cluster administrative domain application provides the capability to maintain a consistent operational environment across multiple systems. A consistent operational environment, along with data and application resilience, is the cornerstone of a highly available application. The operational environment is defined to be the set of resources and attributes that are used by an active application – resources such as user profiles or system values. On IBM i, the cluster administrative domain maintains resources consistently across the nodes in an HA cluster by synchronizing changes to the resources. These resources are called monitored resources within the cluster administrative domain.

Setting Up a Cluster Administrative Domain in 6 Easy Steps
I’m going to show you how to manage your user profiles across multiple systems using the CL commands provided with the PowerHA SystemMirror for i licensed product. In addition to the individual commands used in my example, you can find these steps as options within the WRKCLU command, or you can use one of the PowerHA Web applications that are integrated into IBM Systems Director Navigator for i.

1. Set the QRETSVRSEC System Value

It’s important that the QRETSVRSEC system value be set to 1 before adding any user profiles to your cluster administrative domain. In addition, you should change the password for any user profile you want managed by the administrative domain after changing the system value. This allows the cluster administrative domain to synchronize the password across the nodes in the domain. To change the system value, use this command:

CHGSYSVAL SYSVAL(QRETSVRSEC) VALUE('1')

2. Create the Cluster

Now we’re going to create a one-node cluster and start the node. The cluster name is a 10-character name of your choosing. It’s recommended that the eight-character node name be the same as your system name, for easier management. Using this command, the cluster MYCLUSTER will get created on your first system, represented by the IP address on that system, with a node name of NODEA and the node will be started:

CRTCLU CLUSTER(MYCLUSTER) NODE((NODEA ('9.5.114.142')))

3. Add a Second Node to the Cluster

Next we’re going to add a second node to the MYCLUSTER. This command will result in a cluster node called NODEB, represented by its IP address, being created and started on your second system.

ADDCLUNODE CLUSTER(MYCLUSTER) NODE(NODEB ('9.5.111.151'))

You can repeat this command to add more nodes to your cluster.

4. Create the Cluster Administrative Domain

Now we’re going to create a cluster administrative domain on the two nodes in our cluster using this command:

CRTCAD CLUSTER(MYCLUSTER) ADMDMN(MYDOMAIN) DMNNODL(NODEA NODEB)

5. Start the Cluster Administrative Domain

Next we’ll start the cluster administrative domain, which starts the synchronization process.  Since we don’t have any monitored resources in our administrative domain yet, there’s nothing to synchronize yet. By starting the domain first, any resources we add as monitored-resource entries will get synchronized as they get added. We’ll use the following command to start the domain MYDOMAIN in cluster MYCLUSTER.

STRCAD CLUSTER(MYCLUSTER) ADMDMN(MYDOMAIN)

6. Add a User Profile to the Administrative Domain

Finally, we need to add our user profile to the cluster administrative domain as a monitored resource. The user profile must exist on the system where we’re running the ADDCADMRE command. It may or may not exist on the other node. If it doesn’t exist on the other node, the cluster administrative domain will create it, based on the user profile from the originating system.
   For our example, we only want the administrative domain to synchronize our password, so we will specify PASSWORD as the attribute to monitor. 

ADDCADMRE CLUSTER(MYCLUSTER) ADMDMN(MYDOMAIN)
                        RESOURCE(MYPROFILE)
                        RSCTYPE(*USRPRF) ATTRIBUTE(PASSWORD)       


I now have an active cluster with an active cluster administrative domain monitoring for password changes to MYPROFILE. This means that any time the password is changed on any node, it will be changed on all nodes in the administrative domain. If I change some other attribute of the user profile, such as its job description, the new job description won’t be propagated and changed on the other nodes. I’m not replicating the complete *USRPRF object, only those attributes that I want synchronized; in this case, just the password. I could specify *ALL for the ATTRIBUTE parameter to synchronize all of the supported attributes for user profiles.

You’re now ready to test drive your cluster administrative domain setup. Change the password for your user profile on either one of the nodes in your domain. Now go to second system and sign on with new password. There you go! You have a synchronized password in six simple steps!

There are a couple of neat features with this application that may not be obvious. First of all, you don’t need a central site from which to manage your passwords. Once you have the cluster administrative domain set up, you can make changes from any node, and they will be propagated to the other nodes. Second, and probably even more important, you don’t have to have all of your nodes active in order to maintain consistency. If one or more of the nodes in your domain aren’t active when you make a change to the monitored resource, the change will be applied on the inactive node when it rejoins the cluster.

If you’re interested in reading more about IBM PowerHA SystemMirror for i and the cluster administrative domain application, here are some useful links:

• PowerHA

• Implementing PowerHA for i

• Information Center