Implementing a Shared Storage Pool

I wrote about shared storage pools (here and here) back in January. Recently, I had an opportunity to implement one with a customer.

We had two 720 servers, each of which had two VIO servers. We upgraded to the latest VIOS code, making sure our HMC and firmware were at current levels. Then we presented the LUNs from our SAN, following the steps outlined in my January posts.

First we made sure all the LUNs were set to no reserve.

            chdev -dev <device name> -attr reserve_policy=no_reserve

Then we created the cluster. While I'm giving the names referenced in Nigel Griffiths' presentation (see my first post linked above), for the record, we used our own names.

            cluster -create -clustername galaxy -repopvs hdisk2
            -spname atlantic -sppvs hdisk3 hdisk5 -hostname bluevios1.ibm.com

With that accomplished, I could see that we had a cluster.

            cluster –list

            cluster -status -clustername galaxy

In our cluster, we used one 20G repository LUN and two 500G LUNs for our data.

The cluster –create command took a few minutes to run. On our first try we didn't have fully qualified hostnames on our VIO servers, so we got an error when we tried to create the cluster. Changing the names was easy enough, and after that, the cluster was successfully created.

We ran the cluster –list and cluster –status commands again and got the output we expected. Then from the same node we ran the cluster –addnode command to add a second VIOS to our cluster.

            cluster –addnode –clustername galaxy –hostname redvios1.ibm.com

It took about a minute to add that node, and it was successful. We ran cluster –status again to confirm that the second VIOS was added.

One thing I liked about the process is that the output provides the node name, machine type and model information. This way it's easy to see determine which physical server is running the command.

We did the same procedure for the next two VIO servers. This took a bit longer, likely because they were on another physical server. Still, at the end of the procedure the cluster -status command displayed all four VIO servers in the cluster. When we logged into each of the other VIO servers and ran cluster –status, we saw the same output.

(Note: Running lspv won't tell you that the disks in your storage pool are in use, but the lspv -free command will give you this confirmation. This could be an issue if you were mapping the entire hdisk to a client LPAR -- i.e., the "old" way. But because you're not actually mapping hdisks directly, this isn't necessarily a problem.)

To create a new vdisk to map to our client LPAR, we ran:

            mkbdsp -clustername galaxy -sp atlantic 16G-bd vdisk_red6a -vadapter vhost2

Once we had our disk created and mapped, we ran:

            lssp –clustername galaxy –sp atlantic –bd

That showed us that vdisk_red6a was in the cluster.

Then we ran this command to map it in vios2:

mkbdsp -clustername galaxy -sp atlantic -bd vdisk_red6a -vadapter vhost2

If you compare the command that creates the vdisk to the one that maps the vdisk to the client LPAR, the only difference is the size you provide. Someone can tell me if there's an easier way to do it. For my own amusement I tried using the old mkvdev command. It didn't work.

When we ran lsmap –all, we could see the same vdisk presented to the client, going from both VIO servers.

We then wanted to try live partition mobility using shared storage pools. This posed some problems, but searching on the error message we encountered (HSCLA24E) turned up this entry:

"This week we were trying to migrate some VIO hosted LPARs using XIV disk from one POWER7 system to another. The disk is hosted on a VIO server via the fabric, then using VSCSI devices to map up to the servers. Unfortunately the migration failed and the message we got was HSCLA24E: The migrating partition's virtual SCSI adapter 2 cannot be hosted by the existing virtual I/O server (VIOS) partition on the destination managed system. To migrate the partition, set up the necessary VIOS hosts on the destination managed system, then try the operation again.

"So we did some searching and found the following:

"HSCLA24E error:         

1) On the source VIOS partition, do not set the adapter as required and do not select any client partition can connect when you create a virtual SCSI adapter (can cause the error code).
2) Max transfer size of the used hdisk may not be different on source and destination VIOS.
3) The VIO servers may not have set the reserve_policy to single_path, no_reserve is required.
4) Destination VIO servers are not able to see the disks the client needs.
5) The same VTD (virtual target devices) names may not exist on the destination system."

In our case we addressed no. 1 by unselecting the "any client can connect" option and mapping to the specific client we were using. With these changes, we could successfully migrate the LPAR.

In the course of changing the adapters, we rebooted the VIO servers. Be patient when rebooting. It seems to take some time for the servers to restart and join the cluster. You'll know it's ready when the cluster –status command changes from "state down" to "state OK." (We joked that you only have to give it until "despair + 1.")

Also, be sure to run df and check your /var/vio/SSP/'clustername' filesystem that gets created on all the members of your cluster. That was a quick and dirty way for us to determine that our status was about to change to OK. As the cluster comes online, and as you run cluster –status, you'll see the filesystems mount and the status change from down to OK.

This initial build-out of shared storage pools offers some advantages. For starters, there are fewer, larger LUNs to present and zone to the VIO servers. With larger LUNs being carved up in the pool, there's less to manage with no reserves and the mkvdev commands. Of course some would argue that this advantage is offset by the need to run mkbpsp commands on both VIO servers.

It's also nice being able to login to one cluster node, create a vdisk and see that new vdisk show up on all four nodes, rather than having to login to each VIO server separately. This just feels like a cleaner disk management solution.

As I continue to work with shared storage pools, I'm sure I'll have more lessons to pass along. If you've been using this technology, please share your thoughts in Comments.

Another Grab Bag

As I've noted before, I love passing along tips and tricks. And I love hearing IT horror stories. A little of both this week:

* First, here's an email I got from someone who changed the xfer_size option on his machine. In his words:

“I found this blog [post] a day or two too late. I had tried some AIX fibre card tuning on my Domino servers, which consist of two physical and two VIO NPIV virtual adapters.

"I meant to only change the real cards to 200000 for the xfer_size option, but I had changed the virtual adapters as well, and rebooted. The LPAR hung at LED code 554, and I had to mount OS disk in maintenance mode to mess around with it. This allowed me to undo my change and rmdev the OS disk and paths, to get the LPAR back.

"FYI in case anyone tries this in the future, hopefully they will learn from my mistakes. I am not sure if this is an XIV or an NPIV issue, but I would advise people to not mess with the NPIV xfer_size settings, especially for their root disk."

* I found this item on a mailing list (from Phil L.) and feel it's worth sharing:

"[VIOS 2.2.1.3 introduced] clustering... which is controlled by System Director. System Director automatically starts snmpd, so even if you have it disabled with the viosecure commands it will still start via System Director. The workaround:

            dirsnmpd (Systems Director) is started from:

                /opt/ibm/icc/cimom/bin/startdirsnmpd

            To inhibit dirsnmpd at bootup:

            Edit: startdirsnmpd script

            Comment-out:

                # /usr/bin/startsrc -s snmpd > /dev/null 2>&1

"IBM Support is considering modification of the viosecure rules."

* I saw this in a recent e-mail:

"We have two VIO servers that need to be updated from version 1.4.1.2-FB-9.2 to version 2.2.0.13-FP24 SP03. We did update the two VIO servers in our second data center (so it was a non-production environment). The problem was that we did it incorrectly.

"We put in the migration CD and ran 'updateios; instead of booting off the CD and running 'migrateios.'

"We had to rebuild the whole environment from scratch. We definitely want to avoid that in the production environment!"

Me now. All of you have a test lab to try things out first, right?

"I did have a backup of the VIO servers (created via the command 'ioscli backupios –file … -mksysb,' run as padmin) but we were unable to recover from that backup.

"We worked with IBM Support and they still were not able to recover from our backup so we had to do a fresh install of the VIO server software and rebuild the environment from scratch, and then I had to
recover each LPAR (12) from their mksysb’s. It was a 40-hour weekend!

Me again. All of you know your backups are good and you've tested your recovery process, right?

"In fairness, a lot of that time was actually waiting for CDs to spin. I think if the migration had been done correctly there would only have been a handful of commands that actually needed to be run."

VIOS and IBM i

Two questions for IBM i shops: Are you reluctant to use a VIO server and attach it to your SAN, even though your SAN isn't supported directly by IBM i? Do you end up telling yourself that internal disks give you better performance?

If so, this document might help alleviate your fears.

It covers different topics related to IBM i virtualization and open storage, including how to use an IBM i partition to host another IBM i partition:

"An IBM i 6.1/7.1 LPAR can host one or more additional IBM i LPARs, known as virtual client LPARs. Virtual client partitions typically have no physical I/O hardware assigned and instead leverage virtual I/O resources from the host IBM i partition. The types of hardware resources that can be virtualized by the host LPAR are disk, tape, optical and networking."

There's also this about IBM i using open storage as a client of the VIOS:

"IBM i virtual client partitions can also be hosted by VIOS. VIOS is virtualization software that runs in a separate partition with the purpose to provide virtual storage, optical, tape and networking resources to one or more client partitions. The most immediate benefit that VIOS brings to an IBM i client partition is the ability to expand its storage portfolio to use 512-byte/sector open storage. Open storage volumes (or logical units, LUNs) are physically attached to VIOS through a FC or a Serial-attached SCSI (SAS) connection and then made available to IBM i. While IBM i does not directly attach to the storage area network (SAN) in this case, as soon as open storage LUNs become available through VIOS, they are managed the same way as integrated disks or LUNs from a directly attached storage system and run IBM i on a Power blade."

Finally, something about blades:

"The third major virtualization enhancement with IBM i 6.1 is the ability to run an IBM i LPAR and its applications on a Power blade server, such as IBM BladeCenter JS12 or JS22. Running IBM i on a Power blade is beyond the scope of this paper. Refer to the IBM i on a Power Blade Readme First for a complete technical overview and implementation instructions."

The document covers supported configurations and concepts to help you visualize what I'm proposing. I'll highlight this section on 5.2 performance:

"When creating an open storage LUN configuration for IBM i as a client of VIOS, it is crucial to plan for both capacity and performance. As LUNs are virtualized for IBM i by VIOS instead of being directly connected it may seem that the virtualization layer will necessarily add a significant performance overhead. However, internal IBM performance tests clearly show that the VIOS layer adds a negligible amount of overhead to each I/O operation. Instead, the tests demonstrate that when IBM i uses open storage LUNs virtualized by VIOS, performance is almost entirely determined by the physical and logical configuration of the storage subsystem.

"The IBM Rochester, MN, performance team has run a significant number of tests with IBM i as a client of VIOS using open storage. The resulting recommendations on configuring both the open storage and VIOS are available in the latest Performance Capabilities Reference manual (PCRM)."

I find more customers that are willing to give VIOS a try. I've yet to find one that decided to switch back because performance was unacceptable.

I realize that this blog's readership is very AIX-centric, but plenty of shops run Linux and IBM i as well. It's nice to know that the frame that you're virtualizing with VIOS to run AIX can run other operating systems as well. Not that this is a new idea.

Relatedly, this survey is intended to uncover some misconceptions about the IBM i platform.

 

Automatically Changing IP Addresses in a D/R Environment

I recently spoke to a customer that has its primary and backup servers in different locations. The customer boots from a SAN, with the SAN replicating from site 1 to site 2. In the event of a disaster, the customer wants to fire up its site 2 LPAR from the replicated copy of rootvg. However, the networks are also in different locations.

Rather than do some admin kung fu to allow each network to have the same IP address when it boots up, the customer sought the capability to easily change the IP address depending on the frame being used to boot the LPAR. The customer says this functionality is available in VMware's recovery management product, and wanted to know if the same type of thing can be done from the HMC.

I checked with a couple of my IBM contacts to see if they had any ideas. Chris Gibson had a good one.

"You could write a script that lives on the source system. It checks the system ID (lsattr -El sys0 -a systemid) when the LPAR boots. And if it's a particular system serial number, it could bring up the
interface with a different IP address."

I forwarded this suggestion to the customer, and literally within a day their script was working. With the customer's permission I'm sharing it here, along with their caveat:

"It works … the script is pretty rough. I'm no shell script expert by any means, but it does what I need it to do. I have it in /etc/inittab right before the rctcpip stuff, and that seems to work fine."

Before trying to use this script, make sure your domain, nameserver, gateway and primary and backup server information are accurate for your environment. Of course you might be able to simplify or improve what's here, but this should help you get started. Also note that in the process of posting this script to the blog, some of the formatting may be altered. You savvy scripters should move things around if need be.

#!/bin/ksh
# This script checks to see whether the system is booting off hardware
#at the primary or backup site and sets the IP, gateway, and name
#server based on what hardware it is booting from

# check to see which hardware is booting
OPTION=`lsattr -El sys0 -a systemid -F value`
IPADDRESS=`lsattr -El en0 -a netaddr -F value`
HOST=$(hostname)
DOMAIN="mydomain.com"
PRIMARY="IBM,123"
BACKUP="IBM,456"
NAMESERVER="10.9.0.1"
GATEWAY="10.9.16.1"
# set the primary and backup IP to the correct subnet (xx.1 for primary, xx.2 for backup)
PRIMARYIP=`echo $IPADDRESS | awk -v pos=4 -v repl=1 '{print
substr($0,1,pos-1) repl substr($0,pos+1)}'`
BACKUPIP=`echo $IPADDRESS | awk -v pos=4 -v repl=2 '{print
substr($0,1,pos-1) repl substr($0,pos+1)}'`
BACKUPGW="10.9.16.1"
BACKUPNS="10.9.30.32"

echo "Host Hardware: $OPTION"
echo "Current IP: $IPADDRESS"
echo "Primary IP: $PRIMARYIP"
echo "Backup IP: $BACKUPIP"

if [ "$OPTION" = "$PRIMARY"  ]
then
        echo "Running from primary site"
        if [ "$IPADDRESS" = "$BACKUPIP" ] ; then
                echo "Setting IP for primary location"
                /usr/sbin/mktcpip -h $HOST -a $PRIMARYIP -m
255.255.255.0 -i en0 -n $NAMESERVER -d $DOMAIN -g $GATEWAY -A no -t
N/A
        fi
fi
if [ "$OPTION" = "$BACKUP"  ]
then
        echo "Running from backup site"
    if [ "$IPADDRESS" = "$PRIMARYIP" ] ; then
        /usr/sbin/mktcpip -h $HOST -a $BACKUPIP -m 255.255.255.0 -i en0 -n
$BACKUPNS -d $DOMAIN -g $BACKUPGW -A no -t N/A
    fi
fi

As always, I love getting reader questions and submissions. That includes scripts. Please send me your scripts or any other useful tips. We all benefit when you share your expertise.

 

 

ASO the First Phase in Autonomic Tuning

I've touched on Active System Optimizer (ASO) before, but now that Nigel Griffiths has released an ASO video, it seems an appropriate time to expound on this topic.

To run ASO, you must be at AIX7.1 TL01 or greater on POWER7 hardware running in POWER7 mode. ASO is installed by default on AIX 7. It's not supported on older AIX releases or older hardware. (It will appear to be running, but will actually be hibernating.)

Nigel recommends running:

* oslevel –s to verify your AIX version

* lslpp –L | grep –I optimi to help verify that the ASO fileset is installed

* lsconf | grep ^Processor to verify that your LPAR is running in POWER7 mode.

ASO works under the covers; you don't need to do anything to start it. It optimizes workloads based on AIX kernel statistics and the POWER7 hardware performance counters. It's designed to improve cache and memory affinity by dynamically evaluating tuning options and making changes -- including moving processes and memory -- on the fly. It conducts pre- and post-monitoring to ensure that the changes improve performance. If improvement isn't detected, ASO backs out the changes, hibernates and tries again later. Listen to Nigel's presentation for details on this.

ASO provides cache affinity, aggressive cache affinity and memory affinity. It monitors performance and detects situations where threads can be moved from one chip to another to utilize the closer L3 cache.

ASO operates best on multi-threaded workloads. The jobs it monitors should be stable and long-running so it can most effectively make changes to the workload. It also needs to be a busy LPAR, otherwise you don't gain much by trying to move things around. By "busy," I mean that the processes need at least a 10-second lifetime. If applications come and go more quickly, ASO cannot make recommendations. If you've done manual tuning, either on your own or based on recommendations from IBM Support, ASO will simply hibernate rather than monitor and override your changes. In addition, specific processes can be marked as "don't bother these," and ASO won't impact them.

ASO runs as an SRC kernel service. To locate it, search your machine for the aso process.

Use these commands to get ASO running:

1) start the kernel service with startsrc –s aso

2) run asoo –o aso_active=1

According to Nigel, the aso process uses very minimal CPU time, so this shouldn't add much additional overhead on the system.

Logs are located in the /var/log/aso/* directory. You'll see two files:

* aso.log, which has on/off/hibernating information

* aso_process.log, which provides details of actions and modified processes

Nigel says this log file isn't formally documented, but you should be able figure out what it's doing.

The man page says ASO can run outside of SRC, but this should probably only be done for debugging. You can also set shell variables before starting processes and apps. This provides some control over how they function with ASO.

As always, Nigel has much more than I can cover here. For instance, he shows real life examples of ASO running on his machine, output from the logfile, and more.

To wrap up, he tells us that ASO is largely set and forget. ASO uses near zero CPU when running, and it gently applies changes, tests behavior and undoes the changes if necessary. Because ASO is good for complex, multithreaded, long-running applications, it can move things around inside of your LPAR if they're spread across CPUs.

Best of all, this is, as he says, just the first phase of clever autonomic affinity tuning. So keep your eyes open for what's ahead.

A Good Look at PowerHA

Another great Virtual User Group webinar recently took place, this one featuring Shawn Bodily's presentation, "Introduction to PowerHA SystemMirror for AIX Standard Edition." Be sure to get the presentation materials and listen to the replay. And look forward to the next VUG webinar, when this topic will be continued.

This material is very similar to the presentation on the same topic at last fall's IBM Technical University. That's an indication of the interest surrounding this solution. As Shawn notes in his presentation, IBM has unveiled 23 major releases of the product, an average one release per year. More than 12,000 customers use it worldwide. While Standard Edition is the focus of Shawn's presentation, he points out that PowerHA SystemMirror for AIX Enterprise edition allows for multi-site cluster management, while also including all the functionality of Standard edition.

Shawn notes that "PowerHA gives you cluster management for the data center. The software monitors, detects and reacts to events. It establishes a heartbeat between the systems, and enables automatic switch-over." He then defines what HA solutions generally try to do, which is eliminate single points of failure. The goal is to reduce downtime, but HA can also help you with planned downtime as well. It might not be fault tolerant, but it will be fault resistant.

One chart shows how failure points can be eliminated. To eliminate node failure, use multiple nodes. To eliminate VIO failover, use dual VIO servers. To eliminate site failure, deploy additional sites. Other items are covered, but again, the idea is to build in redundancy so you can continue to provide access to the applications and data that the business runs on.

Shawn discussed his customer interactions (an area I've touched on previously):

"First, another IBM representative will tell the customer about the hardware and the systems' reliability, availability and serviceability (RAS) features. Then a second rep will discuss live partition mobility and how it seamlessly shifts logical partitions from one frame to another. So after 20-30 minutes of hearing about how the hardware never fails, THEN Shawn must step in and explain why the customer should be concerned with high availability and disaster recovery. That's one tough act to follow."

Some additional slides introduce Live Partition Mobility and explain how it allows you to move your live running operating system and application from one physical frame to another. Of course, that's a hardware maintenance solution. What about software maintenance? With PowerHA, you can fail over to the other node, upgrade your application or your operating system, then fail back and do the same thing on the other side. Shawn notes that basically, PowerHA performs the functions that LPM doesn't. He also gets into how PowerHA is used to recover from node failures, network failures, loss of shared storage access, and -- with version 7.1 based on Cluster Aware AIX technology -- rootvg errors. Finally, Shawn covers the differences between PowerHA 6.1 and 7.1.

Shawn makes these additional points:

* Remember, we can fail over from one system to one system, from one system to any other system, from any system to one, and from any to any. We can also fail over between different versions of hardware from the same or different families, assuming you can live with the performance degradation, and after you verify that the version of the operating system you want to run will work with your particular configuration. Failing POWER7 to POWER6 or POWER5 could conceivably work as long as you verify that your particular setup is supported.

* Often you'll want to make your service IP address highly available, along with your application server and your shared storage.

* You can create user defined resources, custom resource groups and more granular resource group options. You can set up resource group dependencies. In his example, you might want to be sure your database is running before you try to start application servers, so you could configure that as a dependency.

* You can configure different priorities and choose which nodes to your resource starts on. This provides a great deal of flexibility and control when setting things up.

* You can also configure things to automatically run DLPAR and COD operations. So you could have a very "skinny" standby node, but when needed it could perform operations on the HMC and bring additional memory and CPU resources online.

* You can have application monitors so you can take actions if PowerHA detects that the application has gone down, and you can set up file collections to have the software help you keep config files or other important files kept in sync across the cluster. This is meant to support all regular files, as opposed to things like trying to keep password files in sync.

* Configuration and smart assistants are available to configure clusters. System Director plugins are also available, both for managing and monitoring your cluster's state.

* The CAA command -- /usr/sbin/clcmd -- can be used to distribute commands across all cluster nodes.

* A cluster test tool can be used to validate clusters. This is also a good way to run tests across many different clusters in the environment, to ensure that we're running the same tests across all of the machines.

Shawn's final slide lists these great resources:

* IBM Redbook: PowerHA SystemMirror 7.1 for AIX

* PowerHA & Aix Support & Compatibility Matrix

* PowerHA Hardware Support Matrix

Incidentally, you can follow Shawn on Twitter.

I really enjoy these webinars, and of course the availability of the replays and presentation materials is a huge convenience. Hopefully my writing about VUG webinars encourages you to take the time and listen for yourself.

Note: Completely unrelatedly, I'm a YouTube sensation. Well, maybe not, but I'm there. Five new videos just went live where you can see me and some other IBM Power Champions talking with IBM's Ian Jarman at last fall's Technical University.

 

 

 

One Guide to the IVM, HMC and SDMC

The just-published IBM Redpaper, "IBM PowerVM Getting Started Guide," shows you how to use the Integrated Virtualization Manager (IVM), Hardware Management Console (HMC) and the Systems Director Management Console (SDMC) to configure your systems. It's an extremely valuable guide that's brief enough, at 104 pages, to be read quickly.

The chapters are independent, so they can be read in any order. I'll run down some highlights in posts over the next two weeks:

Chapter 1

* There's a great chart on page 2 that compares and contrasts the advantages and disadvantages for the IVM, HMC and SDMC. 

* Section 1.2 covers planning:

"Be sure to check system firmware levels on your power server and HMC or SDMC before you start. Decide if you will use Logical Volume Mirroring (LVM) -- in AIX LPARs -- or Multipath IO (MPIO) at the VIOS level. Obviously if you are running NPIV you would want to run MPIO at the AIX level. The examples in this paper use MPIO. Make sure your Fibre Channel switches and adapters are N_Port ID Virtualization (NPIV) capable if you will be using NPIV.

Make sure your network is properly configured.

Check the firewall rules on the HMC or SDMC.

Plan how much processor and memory you will assign to the VIOS for best performance."

* The authors recommend using a dual VIOS architecture -- two VIO servers -- to provide serviceability and scalability. So do I.

* Part of planning includes establishing a VIO slot number scheme. While the SDMC automates slot allocation, the authors illustrate their preferred scheme in Figure 1-2 on page 5.

The authors suggest a VIO slot numbering scheme where the server slot is 101, 102, 103, etc. in both VIO servers, and the client is 11, 12 connecting to VIO1, and 21 and 22 connecting to VIO2. When mapped, VIO1 would map 11 to 101, 12 to 102, and VIO2 would map 21 to 101, 22 to 102. I prefer a numbering scheme where my even-numbered adapters come from one VIOS (VIO1) and my odd-numbered adapters come from the other (VIO2), with both client and server using the same numbers. In my case I like 100, 110, 120, 130 coming from VIO1, and 101 111 121 131 coming from VIO2. Of course, you may have your own numbering scheme -- which I'd love to hear about in Comments.

* Section 1.3 covers the terminology differences between Power- and x86-based systems, which can be handy for someone with little or no background managing power systems. This can help them make the transition in terminology between the two.

* Section 1.4 lists some prerequisites for setting up the machines:

"Check that:

• Your HMC or SDMC (the hardware or the virtual appliance) is configured, up, and running.

• Your HMC or SDMC is connected to the new server's HMC port. We suggest either a private network or a direct cable connection.

• The TCP port 657 is open between the HMC/SDMC and the Virtual Server in order to enable Dynamic Logical Partition functionality.

•    You have IP addresses properly assigned for the HMC, and SDMC.

•    The Power Server is ready to power on.

All your equipment is connected to 802.3ad capable network switches with link aggregation enabled. Refer to the Chapter 5: Advanced Configuration on page 75 for more details.

Fibre Channel fabrics are redundant. Refer to Chapter 5: Advanced Configuration on page 75 for more details.

Ethernet network switches are redundant.

SAN storage for virtual servers (logical partitions) is ready to be provisioned."

Chapter 5

 

The next three chapters are devoted to the specific approaches you might choose to take. Chapter 2 covers the IVM, Chapter 3 the HMC and Chapter 4 the SDMC. I'll dissect those options next week. For now I'll briefly discuss Chapter 5 (Advanced Configuration):

"This chapter describes additional configurations to a dual Virtual I/O Server (VIOS) setup and highlights other advanced configuration practices. The advanced setup addresses performance concerns over the single and dual VIOS setup.

This chapter includes the following sections:

• Adapter ID numbering scheme
• Partition numbering
• VIOS partition and system redundancy
• Advanced VIOS network setup
• Advanced storage connectivity
• Shared processor pools
• Live Partition Mobility
• Active Memory Sharing
• Active Memory Deduplication
• Shared storage pools"

* Table 5-1 illustrates an example of virtual SCSI adapter ID allocations.

* Section 5.4 covers advanced VIOS network setup, including link aggregation and VLAN tagging:

"The VIOS partition is not restricted to only one SEA adapter. It can host multiple SEA adapters where:

A company security policy may advise a separation of VLANs so that one SEA adapter will host secure networks and another SEA adapter will host unsecure networks.

A company may advise a separation of production, testing, and development networks connecting to specific SEA adapter configurations.

"There are considerations regarding the use of IEEE 802.3ad Link Aggregation, 802.1Q VLAN

tagging, and SEA:

There is a maximum of 8 active ports and 8 standby ports in an 802.3ad Link Aggregation device.

Each of the links in a 803.3ad Link Aggregation device should have their speeds set to a common speed setting. For example, set all links to 1g/Full duplex.

A virtual Ethernet adapter is capable of supporting up to 20 VLANS (including the Port Virtual LAN ID - PVID).

A maximum of 16 virtual Ethernet adapters with 20 VLANS assigned to each adapter can be associated to an SEA adapter.

A maximum of 256 virtual Ethernet adapters can be assigned to a single virtual server, including the VIOS partitions.

The IEEE 802.1Q standard supports a maximum of 4096 VLANS. SEA failover is not supported in IVM as it only supports a single VIOS partition."

Whether you set up IBM Power Systems all the time or you're just getting started with the platform, this Redpaper is an excellent resource for learning or reviewing the relevant technology and terminology.

More on Shared Storage Pools

Back in 2010 I wrote about the changes that were coming to VIOS. One of those big changes, shared storage pools, is now a reality. This gives admins another option to consider when setting up disks on Power servers.

In larger companies, disk changes are typically implemented by SAN teams with many other responsibilities and, often, different priorities. However, by allocating storage to the servers up front and setting it up in a storage pool, admins can manage shared storage pools. In doing so, we can be more responsive to requirement changes. And with thin provisioning, we can determine the amount disk we actually use on each server. For the first time since the days of internal disks and expansion drawers, disk is back under our control.

Here's how Nigel Griffiths explains shared storage pools:

"The basic idea behind this technology... is that [VIO servers] across machines can be clustered together and allocate disk blocks from large LUNs assigned to all of them rather than having to do this at the SAN storage level. This uses the vSCSI interface rather than the pass through NPIV method. It also reduces SAN admin required for Live Partition Mobility -- you get the LUN available on all the VIOS and they organise access from there on. It also makes cloning LPARs, disk snapshots and rapid provisioning possible. Plus thin provisioning -- i.e., disk blocks -- are added as and when required, thus saving lots of disk space."

Continuing from last week, here's more from Nigel's presentation.

Since shared storage pools are built on top of cluster-aware AIX, the lscluster command also provides more information, including: lscluster –c  (configuration), lscluster –d  (list all hdisks), lscluster –i  (network interfaces), lscluster –s  (network stats).

In the demo, he also discusses adding disk space and assigning it to client VMs. Keep in mind that you cannot remove a LUN from the pool. You can replace a LUN but you can't remove one.

He also covers thin and thick provisioning using shared storage pools and shows you how to conduct monitoring. Run topas on your VIOS and then enter D (make sure it's upper-case) so you can watch the disk I/O get spread across your disks in 64 MB chunks. From there, Nigel covers how to set up alerts on your disk pool. If you're using thin provisioning, you must ensure you don't run out of space.

Nigel also shares his script, called lspool. It's designed to do the work of multiple scripts by presenting all of the critical information at one time instead of running multiple commands:

# lspool list each cluster and for each list its pools and pool details
~/.profile
clusters=`cluster -list | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo "Cluster list: " $clusters
for clust in $clusters
do
pools=`lssp -clustername $clust | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo Pools in $clust are: $pools
for pool in $pools
do
lssp -clustername $clust | sed '1d' | grep $pool | read p size free
totalLU numLUs junk
let freepc=100*$free/$size
let used=$size-$free
let usedpc=100*$used/$size
echo $pool Pool-Size: $size MB
echo $pool Pool-Free: $free MB Percent Free $freepc
echo $pool Pool-Used: $used MB Percent Used $usedpc
echo $pool Allocated: $totalLU MB for $numLUs Logical Units
alert -list -clustername $clust -spname $pool | sed '1d' | grep $pool
| read p poolid percent
echo $pool Alert-Percent: $percent
if [[ $totalLU > $size ]]
then
let over=$totalLU-$size
echo $pool OverCommitted: yes by $over MB
else
echo $pool OverCommitted: no
fi
done
done

Nigel examines snapshots and cloning with shared storage pools, noting that the different commands -- snapshot –create, snapshot –delete, snapshot –rollback and snapshot –list -- use different syntax. Sometimes it asks for a –spname flag, other times it asks for a –sp flag. Pay attention so you know the flags that are needed with the commands you're running. He also demonstrates how some of this management can be handled using the HMC GUI.

The viosbr command is also covered. I discussed it here.

Nigel recommends that you get started by asking the SAN team to hand over a few TB that you can use for testing. Also make sure your POWER6 and POWER7 servers are at the latest VIOS 2.2 level. It's worth the effort. This technology will save time, boost efficiency and increase your overall responsiveness to users.

Finally, here's Nigel's shared storage pools cheat sheet:

1. chdev -dev <device name> -attr reserve_policy=no_reserve
2. cluster -create -clustername galaxy -repopvs hdisk2
-spname atlantic -sppvs hdisk3 hdisk5 -hostname bluevios1.ibm.com
3. cluster –list
4. cluster -status -clustername galaxy
5. cluster –addnode –clustername galaxy –hostname redvios1.ibm.com
6. cluster -rmnode [-f] -clustername galaxy -hostname redvios1.ibm.com
7. cluster –delete –clustername galaxy
8. lscluster –s or –d or –c or –i = CAA command
9. chsp –add –clustername galaxy -sp atlantic hdisk8 hdisk9
10. chsp -replace -clustername galaxy -sp atlantic -oldpv hdisk4 -newpv hdisk24
11. mkbdsp -clustername galaxy -sp atlantic 16G
-bd vdisk_red6a -vadapter vhost2 [-thick]
12. rmbdsp -clustername galaxy -sp atlantic -bd vdisk_red6a
13. lssp -clustername galaxy -sp atlantic -bd
14. lssp -clustername galaxy
15. alert -set -clustername galaxy –spname atlantic -value 80
16. alert -list -clustername galaxy -spname atlantic
17. errlog –ls
18. snapshot -create name -clustername galaxy -spname atlantic -lu LUs
19. snapshot -delete name -clustername galaxy -spname atlantic -lu LUs
20. snapshot -rollback name -clustername galaxy -spname atlantic -lu LUs
21. snapshot –list -clustername galaxy -spname atlantic
22. viosbr -backup -clustername galaxy -file Daily -frequency daily -numfiles 10
23. viosbr -view -file File -clustername Name …
24. viosbr -restore -clustername Name …
25. lsmap -clustername galaxy –all

Take the time to listen to the replay, and you'll learn even more. I highly recommend it.

Getting Started with Shared Storage Pools

The December AIX Virtual User Group webinar featured Nigel Griffiths' discussion of phase 2 of shared storage pools. If you didn't tune in, download the presentation materials and listen to the replay.

The new shared storage pool functionality is enabled with the latest PowerVM 2.2 service pack, and is a feature of PowerVM Standard and Enterprise. If you already have PowerVM, simply download the VIO server fixpack to obtain these new features. (Note: Because this TL is based on AIX 6.1 TL7, your NIM server must be at AIX 6.1 TL7 or AIX 7.1 TL1 to use your NIM server with your VIO server.)

 One thing to note, as Nigel points out in the presentation, is that the most common VIOS storage options have been around for some time:

1) Logical volumes, created from a volume group and presented to client LPARs
2) Whole local disks
3) SAN LUNs
4) File-backed storage, either from a file system on local disk or a file system on SAN disks
5) NPIV LUNs from SAN

Nigel then discusses the newest option: using SAN LUN disks that are placed into a shared storage pool. This new option, he emphasizes, doesn't eliminate any of the other options. It does not portend the death of NPIV. It's just an additional VIOS storage choice we now have.

Listen to the replay or look over the slides to gather Nigel's thoughts on the benefits of shared storage pools. He explains that fibre channel LUNs and NPIV can be complex. They require knowledge of the SAN switch and the SAN disk subsystem. If you need to make changes, it might take your SAN guys awhile to implement them. This can slow overall responsiveness. That's to say nothing of smaller organizations that don't have dedicated SAN guys. Live partition mobility can be tough work if your disks aren't pre-zoned to the different frames.

With a shared storage pool you pre allocate the disk to the VIO servers. Then it's under your control. You can more easily allocate the space to your virtual machines.

POWER6 and POWER7 servers (including blades) are needed to use shared storage pools. At minimum you should allocate a 1 GB of LUN for your repository and another 1 GB of LUN for data, but in order to be useful, in most cases you'll need much larger LUN(s) – think terabytes of disk -- if you plan to do much with it. 

Your VIOS must have the hostname set correctly to resolve the other hostnames. In Nigel's experience he couldn't use short hostnames -- they had to be changed to their fully qualified names.

He also recommends giving your VIOS a CPU and at least 4 GB of memory. "Skinny" VIOS servers aren't advisable with shared storage pools. Currently, the maximum number of nodes is four, the maximum physical disks in a pool is 256, the maximum virtual disks in a cluster is 1,024, and the number of clients is 40. A pool can have 5 GB to 4 TB of individual disks, and storage pool totals can range from 20 GB to 128 TB. Virtual disk capacity (LU) can be from 1 GB to 4 TB, with only one repository disk.

If you played around with phase one, you'll find that many of your limits have been removed. Now you can use shared storage pools for live partition mobility, perform non-disruptive cluster upgrades and use third party multi-pathing software.

You cannot have active memory sharing paging disks on shared storage pool disks.

Nigel covers the relevant terminology (clusters, pools, logical units, etc.).  He also demonstrates how to actually prepare and set up your disks. In a nutshell you must get your LUNs online and zoned to your VIO servers, and you need to set your reserve policy to no_reserve on your LUNs.

After covering the commands for managing clusters -- cluster –create, cluster –list, cluster –status, cluster –addnode, cluster –rmnode and cluster –delete -- he recommends creating the cluster on one of the VIO servers and then adding additional VIO servers to the shared storage pool cluster. From there, you can allocate space to the VM clients.

Next week I'll have more information from Nigel’s presentation, including scripts and cheat sheets. In the meantime, why not upgrade your test machine's VIO servers to the latest level so you can try this functionality?

Protecting Your Data with mirrorios

In the "good old days" of AIX administration, companies had standalone servers, and rootvg lived on internal disks. We always had at least one pair of internal disks, mirroring them to one another. In the event of a disk failure, you'd unmirror the disks and then replace the failing/failed disk. This was usually accomplished on the fly with hot swap disks. Typically the end users never even knew there was a problem.

Those who still run servers on physical internal disks still need to make sure that they're mirrored in the event a disk needs to be replaced. But even with companies that have moved on to virtualization technology, much of this thinking is preserved today with dual VIO servers and virtual SCSI devices. If you lose one physical path to the storage, the VIO client uses the path provided by the redundant VIOS to keep running.

Most customers I work with these days boot their LPARs (I know, I'm supposed to call them virtual servers, but old habits are hard to break) from SAN. The disk protection and physical disk replacement happens on the back end with a SAN team. Although I do see customers where the SAN guy and the AIX admin are the same person, in all cases the data protection occurs behind the scenes as far as AIX is concerned. The hdisk isn't affected as far as the OS knows.

When installing your VIOS and booting it from internal disks, it's still a good idea to mirror that disk to the other internal disk that's assigned to the same bus on the VIOS. With split backplanes and dual VIO servers, this thinking just needs to be taken a step further: be sure to mirror all of the disks on all of your VIO servers, assuming they're booting locally.

To help in this regard, VIOS has a built-in command called called mirrorios. When you run it, you'll be prompted to reboot your machine when the mirror operation completes. However, that can be deferred by simply running the command, mirrorios –defer.

When this command completes, check your bootlist. You'll find that it hasn't been updated with the disk that you just mirrored to. To remedy this, you must manually run a bosboot on the new disk you mirrored to, and then update your bootlist to reflect the change. If you're wondering why mirrorios can't perform both steps, you're not alone. This is supposed to be an appliance, after all.

So what other methods do you use to protect your data? Surely you take mksysb images and backup your system using TSM or some other method. This of course, brings up the familar but important question: Have you tested your restore procedures lately? Are you sure they work?

Finally, have you made sure your VIO servers are set up correctly? As with a high availability cluster, the wrong time to find out that things aren't set up correctly is when you really need them to work.