Restoring Old Data

A customer was looking to restore some data from an old LTO1 tape. The tape was created in 2005 with versions of AIX and Tivoli Sysback that were common back then. Since the customer no longer had hardware that could read the tape, I was asked if I could help retrieve the data.

An LTO3 tape drive can read LTO1 tapes, so I figured I'd look for that hardware rather than try to scrounge up an LTO1 drive. I contacted IBMers Jeff Werner and Pete Dragovich. From their Chicago lab, they use all kinds of leading-edge and vintage hardware and software to conduct customer-initiated projects and training activities. For instance, they recently performed a simulated HMC update (from 5.2.1 to 6.1.3) for a customer that lacked its own test box. They've also recently done a couple of proof of concept (POC) projects, one to simulate failover with IBM's PowerHA high availability solution and the other involving SLES 11 Linux on POWER6. I figured if anyone could help me, it'd be Jeff and Pete.

I picked up the tape from the customer and brought it to the IBM lab. We built a POWER4 server with an old version of AIX and loaded the Tivoli Sysback code along with the tape utilities we needed for our LTO drive.

After getting everything ready, we loaded the tape, ran /usr/sbin/readsbheader –Nd –f/dev/rmt0 –l, and waited for a list of information. Instead, we immediately got errors. The volume label could not be read. After searching online, we discovered we could run commands to move the tape around and try to read some tape labels, but then we got I/O errors.

Fortunately, the customer had made two copies of the backup tapes, so I picked up the second tape. At that point, I was told that they'd used an unusual block size. That turned out to be the key. We tried 0, 512, and 1024, but hadn't thought about 262144. Once we changed the block size, we were able to restore the data and all went as expected.

With the data restored, we copied it to a Windows machine to burn a DVD copy of the data. The customer then loaded the DVD onto their own Windows machine, and sent it to their server via FTP.

Some takeaways from this project: If you're archiving data, either keep the hardware and software that's capable of reading it, or periodically transfer important old data to newer media. And if you use a different block size when writing data, be sure that's documented—preferably on the tape itself.

I enjoy solving problems, and this was no exception. Now if I'm asked to restore an old Sysback tape, I know I can do it—provided Jeff and Pete are again willing to lend me a hand.

Tracking NPIV Connectivity

IBMer Glenn Lehman posted this script on a mailing list, and with his permission I'm posting it here. Glenn offered this introduction and description: "I search for various 4-digit IBM storage types. My example is coded to recognize 2810, 2107,1750 (which translates to XIV, DS8300, DS6800).

"I share this as an intro... because prior to starting, our team was concerned with how we would keep track of all this configuration information, so I wrote and we now use my handy script. It gathers and parses and organizes all the info we wanted to track into neat columns. It's all automatic, and all real-time.

"Our environment was all p570 frames -- and strictly NPIV (no vSCSI). We have dual SAN fabrics and dual VIO servers per frame for redundancy and disk multi-pathing.

"Let me know if you find this useful, or if you make any changes or modifications so we can share it with others."

 

He adds that the following should be customizable for those familiar with ksh scripting.


#!/bin/ksh
#set -x
#@============================================================================
# Script    :  Collect and display FC/Disk/SAN info for given VIOS pair
# Author    :  Glenn Lehman / IBM
#
# History  :  Created: Thu Jan 26 EST 2012
# Modify    :  Jan 30 2012 - add dph VIOS option
#          :  Jan 31 2012 - add ping test to confirm VIOS is active
#          :  Feb 06 2012 - add uniq date suffix to TMP filename
#          :  Feb 09 2012 - add disk type and count to output
#
# Usage    :  Invoke from remote LPAR with NPIV frame parameter
#=============================================================================
#
##########################  Assumptions  ################################
# There is a remote server that can access all VIO-servers & VIO -clients
#  and that remote server is where this script resides and invokes from
# There are 2 VIO-Servers in a given frame
# There is a distributive shell method configured in the environment
# The LPAR profile name is VIO-<hostname>...
# Example: Profile=>  VIO-hostAix61prd
#          Hostname=> hostAix61prd
# Storage types are one of the following:
# XIV - 2810
# DS8300 - 2107
# DS6800 - 1750
# There are dual SAN Fabrics with unique domains, so that the FCIDs
# can pinpoint the fabric
# Example:
# 0x15 Fabric vs. 0x16 Fabric


##########################  Customization Step  #########################
# Assign the distributive shell command and usage options
#
DSH='dsh -w'

function Get_lsmap
{
## capture the full lsmap output for VIO1 ##
$DSH $VIOH1 "/usr/ios/cli/ioscli lsmap -all -npiv" | cut -f2- -d':' > $TMP.map1
## convert to line separated stanzas ##
cat $TMP.map1 | grep -v "^ $" | \
    awk '
    $1 == "VFC" {print $0 "\n \n"}
    $1 != "VFC" {print $0}' > $TMP.sep1

## capture the full lsmap output for VIO2 ##
$DSH $VIOH2 "/usr/ios/cli/ioscli lsmap -all -npiv" | cut -f2- -d':' > $TMP.map2
## convert to line separated stanzas ##
cat $TMP.map2 | grep -v "^ $" | \
    awk '
    $1 == "VFC" {print $0 "\n \n"}
    $1 != "VFC" {print $0}' > $TMP.sep2

cat $TMP.sep1 $TMP.sep2 > $TMP.map_all
return
}

function Show_Usage
{
echo "Requires a single valid parameter that denotes a VIOS pair
using npiv mapping..."
echo
echo "Usage: $(basename ${0}) < ae2 | b12 | e40 | 88f >"
echo;echo
return
}

function Show_Header
{
echo "\t========================================================================================="
echo "\t  $FRAME virtual (fscsi) mapping - $(date)"
echo "\t========================================================================================="
echo "                                          VIOS
VIOC        "
echo "VFCHOST    ID#  Partition            Server,Slot,Phys
Slot,Virt  Virtual WWPN used  Disks    FC/SAN Status"
echo "----------  ---  -----------------    -------------------
---------  ------------------  ------  ----------------------------
"
return
}

function Ping_VIOS
{
## Verify VIOS pings OK from this server
VIOS=$1
/etc/ping -c 2 $VIOS  > /dev/null
PING_RC=$?
if [[ $PING_RC != 0 ]]; then
  echo "ERROR: Remote server ping from nim0 to $VIOS failed.... aborting."
  exit 99
fi
return
}


# ------ MAiN PROGRAM BEGiNS HERE ------ #
if [[ $# -ne 1 ]]; then
  Show_Usage
  exit 99
fi

## Define pretty-print variables
typeset -L11 VFCHOST
typeset -R2  PLPID
typeset -L20 PLPNAME
typeset -L21 SFCINFO
typeset -L10 CFCINFO
typeset -L8 DISKINFO
typeset -L19 PWWPN

VIOPAIR=$1

##########################  Customization Step  #########################
# Assign Server Frame keyword shortcuts for custom parameter input;
# add these to the Show_Usage "Usage" line
# Assign the various VIO-server hostnames and the Server Frame name
# for each "shortcut"
# Example: Use ae2 as shortcut for Server-9117-MMA-SN10D1AE2
#
case $VIOPAIR in
  ae2 | AE2 ) VIOH1=lbvio1-ae2; VIOH2=lbvio2-ae2;
FRAME='Server-9117-MMA-SN10D1AE2';;
  b12 | B12 ) VIOH1=lbvio1-b12; VIOH2=lbvio2-b12;
FRAME='Server-9117-MMA-SN10D1B12';;
  e40 | E40 ) VIOH1=lbvio1-e40; VIOH2=lbvio2-e40;
FRAME='Server-9117-MMA-SN1014E40';;
  88f | 88F ) VIOH1=lbvio1-88f; VIOH2=lbvio2-88f;
FRAME='Server-9117-MMA-SN106988F-NDev';;
  * ) echo "VIOS parm - $VIOPAIR - is invalid; aborting..."
      Show_Usage
      exit 99
esac

TMP="/tmp/get_vfchosts-$(date +%s)"

## confirm VIOS pair is active and ping-able ##
Ping_VIOS $VIOH1
Ping_VIOS $VIOH2

## collect and reorganize the lsmap output ##
Get_lsmap

Show_Header

## build sorted list of vfchosts ##
grep vfchost $TMP.map1  | awk -v vioh=$VIOH1 '{print $0"  "vioh}' >
$TMP.vfchost1
grep vfchost $TMP.map2  | awk -v vioh=$VIOH2 '{print $0"  "vioh}' >
$TMP.vfchost2
cat $TMP.vfchost1 $TMP.vfchost2 | sort -tt -n -k 2,2 > $TMP.vfchost_all

cat $TMP.vfchost_all | while read VFCHOST PHYLOC PARTID NAME1 NAME2 VIOS; do
    PLPID=$PARTID
    LPNAME=$(echo ${NAME1}${NAME2})
    if [[ $LPNAME != "VIO-"* ]]; then
      PLPNAME='unassigned'
      HOSTNAME=''
      VIOS=$NAME1                      # shifts position if less fields in line
    else
      LPNAME=${LPNAME%AIX}
      HOSTNAME=${LPNAME#VIO-}
      PLPNAME=$LPNAME
    fi
    SSLOT=$(echo $PHYLOC | awk -F- '{print $NF}')
    PFC=$(grep -wp $PHYLOC $TMP.map_all | grep -w FC | awk '{print
$2}' | cut -f2 -d':')
    if [[ $PFC != "fcs"* ]]; then
      PFC='none'
    fi
    SFCINFO="$VIOS,$SSLOT,$PFC"
    CSLOT=$(grep -wp $PHYLOC $TMP.map_all | grep VFC | awk '{print
$NF}' | awk -F- '{print $NF}')
    if [[ $CSLOT != "C"* ]]; then
      CSLOT='undefined'
    fi
    VFC=$(grep -wp $PHYLOC $TMP.map_all | grep VFC | awk '{print $3}'
| cut -f2 -d':')
    if [[ -n $HOSTNAME ]]; then
      $DSH $HOSTNAME "lsdev -c disk" | cut -f2- -d':' > $TMP.disk
      DSKCNT=$(grep -c MPIO $TMP.disk | awk '{print $1}')
      ##########################  Customization Step  #########################
      # Assign the possible remote disk types based on their 4-digit
      # machine type
      #
      if ( grep -w 2810 $TMP.disk > /dev/null ); then
        DISKTYP='XIV'
      elif ( grep -w 2107 $TMP.disk > /dev/null ); then
        DISKTYP='DS83'
      elif ( grep -w 1750 $TMP.disk > /dev/null ); then
        DISKTYP='DS68'
      else
        DISKTYP='unkn'
      fi
      DISKINFO="$DISKTYP($DSKCNT)"
      $DSH $HOSTNAME "fcstat $VFC | grep 'Port'"  | cut -f2- -d':' > $TMP.fcst
      WWPN=$(grep 'World Wide Port' $TMP.fcst | awk '{print $NF}')
      FBRC=$(grep 'Port FC ID' $TMP.fcst | awk '{print substr($NF,1,4)}')
      TYPE=$(grep 'Port Type' $TMP.fcst | awk '{print $NF}')
    else
      WWPN='no active flogi  '
      FBRC='....'
      TYPE=''
      DISKINFO='unknown'
    fi
    PWWPN=$WWPN
    CFCINFO="$CSLOT,$VFC"
    PCNT=$(grep -wp $PHYLOC $TMP.map_all | grep Ports | cut -f2 -d':')
    print "$VFCHOST $PLPID  $PLPNAME $SFCINFO  $CFCINFO  $PWWPN
$DISKINFO $PCNT ports logged in $FBRC $TYPE"
done
echo

## Be polite and clean up after yourself ##
rm -f $TMP.*
exit

In the process of posting this script to the blog, some of the formatting may be altered. Make the logical adjustments as need be. And if you have your own handy scripts or tools, please send them to me and I'll share them here.

Another Grab Bag

As I've noted before, I love passing along tips and tricks. And I love hearing IT horror stories. A little of both this week:

* First, here's an email I got from someone who changed the xfer_size option on his machine. In his words:

“I found this blog [post] a day or two too late. I had tried some AIX fibre card tuning on my Domino servers, which consist of two physical and two VIO NPIV virtual adapters.

"I meant to only change the real cards to 200000 for the xfer_size option, but I had changed the virtual adapters as well, and rebooted. The LPAR hung at LED code 554, and I had to mount OS disk in maintenance mode to mess around with it. This allowed me to undo my change and rmdev the OS disk and paths, to get the LPAR back.

"FYI in case anyone tries this in the future, hopefully they will learn from my mistakes. I am not sure if this is an XIV or an NPIV issue, but I would advise people to not mess with the NPIV xfer_size settings, especially for their root disk."

* I found this item on a mailing list (from Phil L.) and feel it's worth sharing:

"[VIOS 2.2.1.3 introduced] clustering... which is controlled by System Director. System Director automatically starts snmpd, so even if you have it disabled with the viosecure commands it will still start via System Director. The workaround:

            dirsnmpd (Systems Director) is started from:

                /opt/ibm/icc/cimom/bin/startdirsnmpd

            To inhibit dirsnmpd at bootup:

            Edit: startdirsnmpd script

            Comment-out:

                # /usr/bin/startsrc -s snmpd > /dev/null 2>&1

"IBM Support is considering modification of the viosecure rules."

* I saw this in a recent e-mail:

"We have two VIO servers that need to be updated from version 1.4.1.2-FB-9.2 to version 2.2.0.13-FP24 SP03. We did update the two VIO servers in our second data center (so it was a non-production environment). The problem was that we did it incorrectly.

"We put in the migration CD and ran 'updateios; instead of booting off the CD and running 'migrateios.'

"We had to rebuild the whole environment from scratch. We definitely want to avoid that in the production environment!"

Me now. All of you have a test lab to try things out first, right?

"I did have a backup of the VIO servers (created via the command 'ioscli backupios –file … -mksysb,' run as padmin) but we were unable to recover from that backup.

"We worked with IBM Support and they still were not able to recover from our backup so we had to do a fresh install of the VIO server software and rebuild the environment from scratch, and then I had to
recover each LPAR (12) from their mksysb’s. It was a 40-hour weekend!

Me again. All of you know your backups are good and you've tested your recovery process, right?

"In fairness, a lot of that time was actually waiting for CDs to spin. I think if the migration had been done correctly there would only have been a handful of commands that actually needed to be run."

Valuable Insight into Storage

I recently took some online training that I found interesting and valuable. The webinar presenter, Norman Bogard, compares and contrasts network-attached storage (NAS) and storage area networks (SANs). (In his presentation Norman acknowledges Brett Cooper and Nils Haustein for their input, so I want to be sure to mention them here.)

When I think of early iterations of direct-attached storage, like SSA disks or regular internal SCSI disks, it amazes me how far we've come. Advances have been brought not just to storage hardware, but to the network infrastucture. Networks and switches are so much faster and more robust now.

The webinar opens with a review of the history of NAS and SAN and protocols like NFS, NCP, SMB and CIFS. Then some terminology is introduced. I'm paraphrasing and borrowing some of the language from the slide deck since Norman did such a good job of compiling the material:

"SAN, or block storage, will leverage Small Computer System Interface (SCSI) commands to read-write specific blocks. Common SCSI access methods include Fiber Channel (FC), Internet Small Computer System Interface (iSCSI), or InfiniBand (IB). InfiniBand is a high speed network interconnect.

"NAS, or file storage, reads and writes files instead of blocks. The NAS has control of the files, contrasted with a SAN where the server would have control over the files.  A file server is a storage server dedicated (primarily) to serving file-based workloads.

"A NAS gateway is a server that provides network-based storage virtualization. It provides protocol translation from host-based CIFS/NFS to Storage Area Network (SAN) based block storage. Examples of NAS gateways are IBM N series & SONAS; NetApp V Series; EMC VNX/Celerra; OnStor (LSI); HP P4000 Unified Gateway.

"Unified Storage is a single logical, centrally managed storage platform that serves both block (FC, iSCSI, IB) and file-based (CIFS, NFS, HTTP, etc.) workloads. Examples of Unified Storage includes IBM N series; NetApp V series; IBM Storwize V7000 Unified.

"When you compare NAS and SAN, you will find that they have similar concepts. For example, your redundancy for your SAN will come from your MPIO or SDD drivers, while redundancy for the NAS will come from teaming or trunking your network ports for resiliency or improved bandwidth, depending on how you have set things up.

"Your security for a SAN will come from LUN Masking and Zoning, while you would control access on the network the same way you always would, with things like VLANS, exports, and shares.

"Your physical connectivity to the SAN would come via the HBA, while your network traffic for the NAS would go out the same NIC that it always has, at least until converged network adapters become more widely deployed. Once we have converged adapters, all of the traffic will be network traffic, although you will then be dealing with more encapsulation of the different frames and protocols.

"Your underlying protocol on a SAN is SCSI, while you use the same IP/UDP protocols that you do with networking when you use NAS.

"You will call your SAN devices arrays, and you will present LUNs, while your NAS will have filers and data movers. You will have structured/relational data on a SAN, and unstructured data on a NAS."

From there, Norman contrasts the concepts of block storage (SAN) and file storage (NAS). I'll share more about this webinar in next week's post.

 

The Importance of DR Testing

Recently my customer wanted to see if its old, unsupported application could be recovered in an emergency. They were running AIX 5.2, and I was cloning that to another piece of older hardware for a disaster/recovery test. While the customer had been taking mksysb and application backups for some time, this was the first actual attempt to recover the system.

After restoring the mksysb to the target machine, we went to run the vendor's built-in scripts to recover the database. It turns out that all of the data and binaries we needed to run the recovery operation was on datavg instead of rootvg.

It also turned out there was no datavg backup, only the database backups. This became the first item to address, ensuring we had a savevg of datavg. In this case the customer had 135 logical volumes that had to be recreated, some jfs2 and some raw.

The customer really wanted this cloned machine to be identical to the source machine, down to the PP size, but no way was I going to recreate 135 logical volumes manually. So I went ahead and did a savevg from the source machine. Had this been an actual DR situation, we would have already been in trouble had the LV information not been stored somewhere. (Hint: Besides backups, it may be handy to have output from important files like /etc/filesystems available to you in case of an emergency.)

When I tried to restore the information from the savevg and remake the volume group

(smitty/system storage management/logical volume manager/volume groups/remake a volume group),

it kept coming up with a 512MB PP size instead of the 64MB PP size I was inputting. Even when I tried it from the command line (restvg –f /dev/rmt0 –r –n –P 64 hdisk4), it'd still create the 512MB PP size.

However, since I still had the source system, it was a simple matter of taking the logical volume information from the source volume group and copying it to /tmp:

lsvg –l datavg > /tmp/datavgout.file

Because I only cared about the first and third columns of the lsvg output, I ran this command to obtain the LV name and LV type:

cat /tmp/datavgout.file | awk '{print $1, $3}' > /tmp/datavgout2.file

I created the datavg on the target system manually with the 64 MB PP size. Then I edited the datavgout2.file and made sure it had the correct LV name, LV type and the number of PPs that I wanted on the target machine. To read the file and create the LVs, I ran this simple loop:

cat '/tmp/datavgout2.file' | while read $i $j $k
do
mklv –t $j –y $i datavg1 $k
done

($i is the name, $j is the type and $k is the number of PPs.)

I did end up using smitty (smitty/system storage management/logical volume manager/volume groups/restore) to restore the files in the jfs2 filesystems.

Once the volume group had been recreated and the necessary files were restored, I could use the database backup tape to restore the database.

The customer now takes a daily savevg of datavg, and all of the necessary LV information from the entire system is saved as part of the rootvg backup. In the end we were able to get a running system. Even more imporant, we learned something. Without going through this exercise, my customer may have been missing key information and data it needed to restore the system in an actual disaster.

This is why simply having a DR plan isn't enough. Your plan must be tested. Even if you think you have good backups, it might not be the case.

Incidentally, Anthony English recently wrote about recovering datavg filesystems as well. He discusses using the mkvgdata (to capture the volume group structure) and restvg commands. His information is worth considering if you find yourself recreating a volume group. It's certainly simpler than going through the gyrations that I went through, though I'll still need to test it to see if Anthony's approach will eliminate my customer's PP issue.

VIOS and IBM i

Two questions for IBM i shops: Are you reluctant to use a VIO server and attach it to your SAN, even though your SAN isn't supported directly by IBM i? Do you end up telling yourself that internal disks give you better performance?

If so, this document might help alleviate your fears.

It covers different topics related to IBM i virtualization and open storage, including how to use an IBM i partition to host another IBM i partition:

"An IBM i 6.1/7.1 LPAR can host one or more additional IBM i LPARs, known as virtual client LPARs. Virtual client partitions typically have no physical I/O hardware assigned and instead leverage virtual I/O resources from the host IBM i partition. The types of hardware resources that can be virtualized by the host LPAR are disk, tape, optical and networking."

There's also this about IBM i using open storage as a client of the VIOS:

"IBM i virtual client partitions can also be hosted by VIOS. VIOS is virtualization software that runs in a separate partition with the purpose to provide virtual storage, optical, tape and networking resources to one or more client partitions. The most immediate benefit that VIOS brings to an IBM i client partition is the ability to expand its storage portfolio to use 512-byte/sector open storage. Open storage volumes (or logical units, LUNs) are physically attached to VIOS through a FC or a Serial-attached SCSI (SAS) connection and then made available to IBM i. While IBM i does not directly attach to the storage area network (SAN) in this case, as soon as open storage LUNs become available through VIOS, they are managed the same way as integrated disks or LUNs from a directly attached storage system and run IBM i on a Power blade."

Finally, something about blades:

"The third major virtualization enhancement with IBM i 6.1 is the ability to run an IBM i LPAR and its applications on a Power blade server, such as IBM BladeCenter JS12 or JS22. Running IBM i on a Power blade is beyond the scope of this paper. Refer to the IBM i on a Power Blade Readme First for a complete technical overview and implementation instructions."

The document covers supported configurations and concepts to help you visualize what I'm proposing. I'll highlight this section on 5.2 performance:

"When creating an open storage LUN configuration for IBM i as a client of VIOS, it is crucial to plan for both capacity and performance. As LUNs are virtualized for IBM i by VIOS instead of being directly connected it may seem that the virtualization layer will necessarily add a significant performance overhead. However, internal IBM performance tests clearly show that the VIOS layer adds a negligible amount of overhead to each I/O operation. Instead, the tests demonstrate that when IBM i uses open storage LUNs virtualized by VIOS, performance is almost entirely determined by the physical and logical configuration of the storage subsystem.

"The IBM Rochester, MN, performance team has run a significant number of tests with IBM i as a client of VIOS using open storage. The resulting recommendations on configuring both the open storage and VIOS are available in the latest Performance Capabilities Reference manual (PCRM)."

I find more customers that are willing to give VIOS a try. I've yet to find one that decided to switch back because performance was unacceptable.

I realize that this blog's readership is very AIX-centric, but plenty of shops run Linux and IBM i as well. It's nice to know that the frame that you're virtualizing with VIOS to run AIX can run other operating systems as well. Not that this is a new idea.

Relatedly, this survey is intended to uncover some misconceptions about the IBM i platform.

 

ASO the First Phase in Autonomic Tuning

I've touched on Active System Optimizer (ASO) before, but now that Nigel Griffiths has released an ASO video, it seems an appropriate time to expound on this topic.

To run ASO, you must be at AIX7.1 TL01 or greater on POWER7 hardware running in POWER7 mode. ASO is installed by default on AIX 7. It's not supported on older AIX releases or older hardware. (It will appear to be running, but will actually be hibernating.)

Nigel recommends running:

* oslevel –s to verify your AIX version

* lslpp –L | grep –I optimi to help verify that the ASO fileset is installed

* lsconf | grep ^Processor to verify that your LPAR is running in POWER7 mode.

ASO works under the covers; you don't need to do anything to start it. It optimizes workloads based on AIX kernel statistics and the POWER7 hardware performance counters. It's designed to improve cache and memory affinity by dynamically evaluating tuning options and making changes -- including moving processes and memory -- on the fly. It conducts pre- and post-monitoring to ensure that the changes improve performance. If improvement isn't detected, ASO backs out the changes, hibernates and tries again later. Listen to Nigel's presentation for details on this.

ASO provides cache affinity, aggressive cache affinity and memory affinity. It monitors performance and detects situations where threads can be moved from one chip to another to utilize the closer L3 cache.

ASO operates best on multi-threaded workloads. The jobs it monitors should be stable and long-running so it can most effectively make changes to the workload. It also needs to be a busy LPAR, otherwise you don't gain much by trying to move things around. By "busy," I mean that the processes need at least a 10-second lifetime. If applications come and go more quickly, ASO cannot make recommendations. If you've done manual tuning, either on your own or based on recommendations from IBM Support, ASO will simply hibernate rather than monitor and override your changes. In addition, specific processes can be marked as "don't bother these," and ASO won't impact them.

ASO runs as an SRC kernel service. To locate it, search your machine for the aso process.

Use these commands to get ASO running:

1) start the kernel service with startsrc –s aso

2) run asoo –o aso_active=1

According to Nigel, the aso process uses very minimal CPU time, so this shouldn't add much additional overhead on the system.

Logs are located in the /var/log/aso/* directory. You'll see two files:

* aso.log, which has on/off/hibernating information

* aso_process.log, which provides details of actions and modified processes

Nigel says this log file isn't formally documented, but you should be able figure out what it's doing.

The man page says ASO can run outside of SRC, but this should probably only be done for debugging. You can also set shell variables before starting processes and apps. This provides some control over how they function with ASO.

As always, Nigel has much more than I can cover here. For instance, he shows real life examples of ASO running on his machine, output from the logfile, and more.

To wrap up, he tells us that ASO is largely set and forget. ASO uses near zero CPU when running, and it gently applies changes, tests behavior and undoes the changes if necessary. Because ASO is good for complex, multithreaded, long-running applications, it can move things around inside of your LPAR if they're spread across CPUs.

Best of all, this is, as he says, just the first phase of clever autonomic affinity tuning. So keep your eyes open for what's ahead.

A Good Look at PowerHA

Another great Virtual User Group webinar recently took place, this one featuring Shawn Bodily's presentation, "Introduction to PowerHA SystemMirror for AIX Standard Edition." Be sure to get the presentation materials and listen to the replay. And look forward to the next VUG webinar, when this topic will be continued.

This material is very similar to the presentation on the same topic at last fall's IBM Technical University. That's an indication of the interest surrounding this solution. As Shawn notes in his presentation, IBM has unveiled 23 major releases of the product, an average one release per year. More than 12,000 customers use it worldwide. While Standard Edition is the focus of Shawn's presentation, he points out that PowerHA SystemMirror for AIX Enterprise edition allows for multi-site cluster management, while also including all the functionality of Standard edition.

Shawn notes that "PowerHA gives you cluster management for the data center. The software monitors, detects and reacts to events. It establishes a heartbeat between the systems, and enables automatic switch-over." He then defines what HA solutions generally try to do, which is eliminate single points of failure. The goal is to reduce downtime, but HA can also help you with planned downtime as well. It might not be fault tolerant, but it will be fault resistant.

One chart shows how failure points can be eliminated. To eliminate node failure, use multiple nodes. To eliminate VIO failover, use dual VIO servers. To eliminate site failure, deploy additional sites. Other items are covered, but again, the idea is to build in redundancy so you can continue to provide access to the applications and data that the business runs on.

Shawn discussed his customer interactions (an area I've touched on previously):

"First, another IBM representative will tell the customer about the hardware and the systems' reliability, availability and serviceability (RAS) features. Then a second rep will discuss live partition mobility and how it seamlessly shifts logical partitions from one frame to another. So after 20-30 minutes of hearing about how the hardware never fails, THEN Shawn must step in and explain why the customer should be concerned with high availability and disaster recovery. That's one tough act to follow."

Some additional slides introduce Live Partition Mobility and explain how it allows you to move your live running operating system and application from one physical frame to another. Of course, that's a hardware maintenance solution. What about software maintenance? With PowerHA, you can fail over to the other node, upgrade your application or your operating system, then fail back and do the same thing on the other side. Shawn notes that basically, PowerHA performs the functions that LPM doesn't. He also gets into how PowerHA is used to recover from node failures, network failures, loss of shared storage access, and -- with version 7.1 based on Cluster Aware AIX technology -- rootvg errors. Finally, Shawn covers the differences between PowerHA 6.1 and 7.1.

Shawn makes these additional points:

* Remember, we can fail over from one system to one system, from one system to any other system, from any system to one, and from any to any. We can also fail over between different versions of hardware from the same or different families, assuming you can live with the performance degradation, and after you verify that the version of the operating system you want to run will work with your particular configuration. Failing POWER7 to POWER6 or POWER5 could conceivably work as long as you verify that your particular setup is supported.

* Often you'll want to make your service IP address highly available, along with your application server and your shared storage.

* You can create user defined resources, custom resource groups and more granular resource group options. You can set up resource group dependencies. In his example, you might want to be sure your database is running before you try to start application servers, so you could configure that as a dependency.

* You can configure different priorities and choose which nodes to your resource starts on. This provides a great deal of flexibility and control when setting things up.

* You can also configure things to automatically run DLPAR and COD operations. So you could have a very "skinny" standby node, but when needed it could perform operations on the HMC and bring additional memory and CPU resources online.

* You can have application monitors so you can take actions if PowerHA detects that the application has gone down, and you can set up file collections to have the software help you keep config files or other important files kept in sync across the cluster. This is meant to support all regular files, as opposed to things like trying to keep password files in sync.

* Configuration and smart assistants are available to configure clusters. System Director plugins are also available, both for managing and monitoring your cluster's state.

* The CAA command -- /usr/sbin/clcmd -- can be used to distribute commands across all cluster nodes.

* A cluster test tool can be used to validate clusters. This is also a good way to run tests across many different clusters in the environment, to ensure that we're running the same tests across all of the machines.

Shawn's final slide lists these great resources:

* IBM Redbook: PowerHA SystemMirror 7.1 for AIX

* PowerHA & Aix Support & Compatibility Matrix

* PowerHA Hardware Support Matrix

Incidentally, you can follow Shawn on Twitter.

I really enjoy these webinars, and of course the availability of the replays and presentation materials is a huge convenience. Hopefully my writing about VUG webinars encourages you to take the time and listen for yourself.

Note: Completely unrelatedly, I'm a YouTube sensation. Well, maybe not, but I'm there. Five new videos just went live where you can see me and some other IBM Power Champions talking with IBM's Ian Jarman at last fall's Technical University.

 

 

 

IVM, HMC and SDMC Continued

Continuing from last week, here's more on the recently released IBM Redpaper, "IBM PowerVM Getting Started Guide."

Chapter 2: IVM

From the authors:

"IBM developed the Integrated Virtualization Manager (IVM) as a server management solution that performs a subset of the HMC and SDMC features for a single server, avoiding the need for a dedicated HMC or SDMC server. IVM manages a single stand-alone server -- a second server managed by IVM has its own instance of IVM installed. With the subset of HMC and SDMC server functionality, IVM provides a solution that enables the administrator to quickly set up a server. IVM is integrated within the Virtual I/O Server product, which services I/O, memory, and processor virtualization in IBM Power Systems.

"There are many environments that need small partitioned systems, either for test reasons or for specific requirements, for which the HMC and SDMC solutions are not ideal. A sample situation is where there are small partitioned systems that cannot share a common HMC or SDMC because they are in multiple locations.

"IVM is a simplified hardware management solution that inherits most of the HMC features. It manages a single server, avoiding the need for an independent personal computer. It is designed to provide a solution that enables the administrator to reduce system setup time and to make hardware management easier, at a lower cost.

"When not using either the HMC or the SDMC, VIOS takes control of all the hardware resources. There is no need to create a specific partition for the VIOS. When VIOS is installed using the default settings, it installs on the server’s first internal disk controller and onto the first disk on that controller. IVM is part of VIOS and activated when VIOS is installed without an HMC or SDMC."

Chapter 2 continues with details on IVM installation.

I wish this chapter would include screen shots. (There are screen shots in chapters 3-4.) The Redpaper describes the steps, but for those unfamiliar with the interface it might be confusing. Some screen shots could help.

Chapter 3: HMC

More from the authors:

"Note: There is flexibility for you to plan your own adapter numbering scheme. The Maximum virtual adapters setting needs to be set in the Virtual Adapters window to allow for your numbering scheme. The maximum setting is 65535 but the higher the setting, the more memory the managed system reserves to manage the adapters."

They cover the three VIOS installation methods: DVD, via the HMC (using the installios command) and via Network Installation Manager (NIM). One of the notes says:

"Interface en5 is the SEA adapter created in 3 on page 29. Alternatively, an additional virtual adapter may be created for the VIOS remote connection, or another physical adapter may be used (it will need to be cabled) for the TCP/IP remote connection. TCP and UDP port 657 must be open between the HMC and the VIOS. This is a requirement for DLPAR (using RMC protocol)."

I know when I set up shared Ethernet adapters on VIO servers, I like to add an additional virtual Ethernet adapter to put my VIO IP address on. This allows me to perform maintenance on my VIOS and SEA without an outage, as the network traffic goes out my backup SEA on my other VIOS.

Section 3.2 covers setting up dual VIO servers:

"The benefit of a dual VIOS setup is that it promotes Redundancy, Accessibility and Serviceability (RAS). It also offers load balancing capabilities for MPIO and for multi SEA configuration setups. The differences between a single and dual VIOS setup are:

The additional VIOS partition
The additional virtual Ethernet adapter used as the SEA Control Channel adapter per VIOS
Setting the trunk priority on the virtual Ethernet adapters used for bridging to physical adapters in an SEA configuration."

The authors explain how to move from a single VIO SEA to a dual VIO scenario by adding the control channel adapter using this command:

   chdev -dev ent5 -attr ctl_chan=ent6 ha_mode=auto

They also mention that we can run commands on the VIO command line or use cfgassist, which is similar to smitty in AIX.

Section 3.3 covers setting up virtual fibre. The authors argue that virtual SCSI disks be used for rootvg and NPIV be used for data LUNs:

"Virtual Fibre Channel allows disks to be assigned directly to the client partitions from the SAN storage system. With virtual SCSI, the disks are assigned to the VIOS partition before they are mapped to a virtual SCSI adapter.

"The preference is to still use virtual SCSI for client partition operating system disk, and use virtual Fibre Channel for the data. The reasons for using virtual SCSI for client partition operating system disks are:

* When the disks are assigned VIOS first, they can be checked before having them mapped to a client. Whereas using virtual Fibre Channel this cannot be determined until the client partition is loaded from an installation source.

* Operating systems such as AIX and Linux have their kernels running in memory. If serious SAN issues are being experienced, the VIOS will first detect the problem and sever the link to the client partition. The client partition will halt abruptly reducing any risk to data corruption. With operating systems using virtual Fibre Channel or physical Fibre Channel, the partition will remain running for a period. During that period the client partition is susceptible to data corruption.

* Operating system disks using virtual SCSI are not reliant on external device drivers whereas operating system disks using virtual Fibre Channel are. When it comes to upgrading the external device drivers, the client partitions would need to follow special procedures to upgrade."

Chapter 4: SDMC

From the authors:

"The IBM Systems Director Management Console (SDMC) provides system administrators the ability to manage IBM Power System servers as well as IBM Power Blade servers. The SDMC organizes tasks in a single panel that simplifies views of systems and day-to-tay tasks. The SDMC is also designed to be integrated into the administrative framework of IBM Systems Director.

"The SDMC can automatically handle the slot allocation of virtual adapters for the user. With the SDMC the user can choose to either let the SDMC manage the slot allocations, or use the traditional manual mechanism to allocate the virtual adapter IDs."

According to section 4.1.4, setting up an SEA failover configuration is a simple GUI operation when using SDMC:

 

 

Select the primary VIOS, the physical adapter you want to use and the backup VIO and its physical adapter. Then hit OK:

"The SDMC automatically creates the SEA adapters on both VIOS1 and VIOS2. The SDMC will also configure the control channel as a part of this step. The virtual Ethernet adapter with the highest VLAN ID is used for the SEA control channel."

This should remove the possibility of errors arising from setting up the control channels manually.

Although you can do the same with the HMC GUI, I still prefer to manage things on the command line.

The publication has much more. It's well worth your time.

One Guide to the IVM, HMC and SDMC

The just-published IBM Redpaper, "IBM PowerVM Getting Started Guide," shows you how to use the Integrated Virtualization Manager (IVM), Hardware Management Console (HMC) and the Systems Director Management Console (SDMC) to configure your systems. It's an extremely valuable guide that's brief enough, at 104 pages, to be read quickly.

The chapters are independent, so they can be read in any order. I'll run down some highlights in posts over the next two weeks:

Chapter 1

* There's a great chart on page 2 that compares and contrasts the advantages and disadvantages for the IVM, HMC and SDMC. 

* Section 1.2 covers planning:

"Be sure to check system firmware levels on your power server and HMC or SDMC before you start. Decide if you will use Logical Volume Mirroring (LVM) -- in AIX LPARs -- or Multipath IO (MPIO) at the VIOS level. Obviously if you are running NPIV you would want to run MPIO at the AIX level. The examples in this paper use MPIO. Make sure your Fibre Channel switches and adapters are N_Port ID Virtualization (NPIV) capable if you will be using NPIV.

Make sure your network is properly configured.

Check the firewall rules on the HMC or SDMC.

Plan how much processor and memory you will assign to the VIOS for best performance."

* The authors recommend using a dual VIOS architecture -- two VIO servers -- to provide serviceability and scalability. So do I.

* Part of planning includes establishing a VIO slot number scheme. While the SDMC automates slot allocation, the authors illustrate their preferred scheme in Figure 1-2 on page 5.

The authors suggest a VIO slot numbering scheme where the server slot is 101, 102, 103, etc. in both VIO servers, and the client is 11, 12 connecting to VIO1, and 21 and 22 connecting to VIO2. When mapped, VIO1 would map 11 to 101, 12 to 102, and VIO2 would map 21 to 101, 22 to 102. I prefer a numbering scheme where my even-numbered adapters come from one VIOS (VIO1) and my odd-numbered adapters come from the other (VIO2), with both client and server using the same numbers. In my case I like 100, 110, 120, 130 coming from VIO1, and 101 111 121 131 coming from VIO2. Of course, you may have your own numbering scheme -- which I'd love to hear about in Comments.

* Section 1.3 covers the terminology differences between Power- and x86-based systems, which can be handy for someone with little or no background managing power systems. This can help them make the transition in terminology between the two.

* Section 1.4 lists some prerequisites for setting up the machines:

"Check that:

• Your HMC or SDMC (the hardware or the virtual appliance) is configured, up, and running.

• Your HMC or SDMC is connected to the new server's HMC port. We suggest either a private network or a direct cable connection.

• The TCP port 657 is open between the HMC/SDMC and the Virtual Server in order to enable Dynamic Logical Partition functionality.

•    You have IP addresses properly assigned for the HMC, and SDMC.

•    The Power Server is ready to power on.

All your equipment is connected to 802.3ad capable network switches with link aggregation enabled. Refer to the Chapter 5: Advanced Configuration on page 75 for more details.

Fibre Channel fabrics are redundant. Refer to Chapter 5: Advanced Configuration on page 75 for more details.

Ethernet network switches are redundant.

SAN storage for virtual servers (logical partitions) is ready to be provisioned."

Chapter 5

 

The next three chapters are devoted to the specific approaches you might choose to take. Chapter 2 covers the IVM, Chapter 3 the HMC and Chapter 4 the SDMC. I'll dissect those options next week. For now I'll briefly discuss Chapter 5 (Advanced Configuration):

"This chapter describes additional configurations to a dual Virtual I/O Server (VIOS) setup and highlights other advanced configuration practices. The advanced setup addresses performance concerns over the single and dual VIOS setup.

This chapter includes the following sections:

• Adapter ID numbering scheme
• Partition numbering
• VIOS partition and system redundancy
• Advanced VIOS network setup
• Advanced storage connectivity
• Shared processor pools
• Live Partition Mobility
• Active Memory Sharing
• Active Memory Deduplication
• Shared storage pools"

* Table 5-1 illustrates an example of virtual SCSI adapter ID allocations.

* Section 5.4 covers advanced VIOS network setup, including link aggregation and VLAN tagging:

"The VIOS partition is not restricted to only one SEA adapter. It can host multiple SEA adapters where:

A company security policy may advise a separation of VLANs so that one SEA adapter will host secure networks and another SEA adapter will host unsecure networks.

A company may advise a separation of production, testing, and development networks connecting to specific SEA adapter configurations.

"There are considerations regarding the use of IEEE 802.3ad Link Aggregation, 802.1Q VLAN

tagging, and SEA:

There is a maximum of 8 active ports and 8 standby ports in an 802.3ad Link Aggregation device.

Each of the links in a 803.3ad Link Aggregation device should have their speeds set to a common speed setting. For example, set all links to 1g/Full duplex.

A virtual Ethernet adapter is capable of supporting up to 20 VLANS (including the Port Virtual LAN ID - PVID).

A maximum of 16 virtual Ethernet adapters with 20 VLANS assigned to each adapter can be associated to an SEA adapter.

A maximum of 256 virtual Ethernet adapters can be assigned to a single virtual server, including the VIOS partitions.

The IEEE 802.1Q standard supports a maximum of 4096 VLANS. SEA failover is not supported in IVM as it only supports a single VIOS partition."

Whether you set up IBM Power Systems all the time or you're just getting started with the platform, this Redpaper is an excellent resource for learning or reviewing the relevant technology and terminology.