Restoring Old Data

A customer was looking to restore some data from an old LTO1 tape. The tape was created in 2005 with versions of AIX and Tivoli Sysback that were common back then. Since the customer no longer had hardware that could read the tape, I was asked if I could help retrieve the data.

An LTO3 tape drive can read LTO1 tapes, so I figured I'd look for that hardware rather than try to scrounge up an LTO1 drive. I contacted IBMers Jeff Werner and Pete Dragovich. From their Chicago lab, they use all kinds of leading-edge and vintage hardware and software to conduct customer-initiated projects and training activities. For instance, they recently performed a simulated HMC update (from 5.2.1 to 6.1.3) for a customer that lacked its own test box. They've also recently done a couple of proof of concept (POC) projects, one to simulate failover with IBM's PowerHA high availability solution and the other involving SLES 11 Linux on POWER6. I figured if anyone could help me, it'd be Jeff and Pete.

I picked up the tape from the customer and brought it to the IBM lab. We built a POWER4 server with an old version of AIX and loaded the Tivoli Sysback code along with the tape utilities we needed for our LTO drive.

After getting everything ready, we loaded the tape, ran /usr/sbin/readsbheader –Nd –f/dev/rmt0 –l, and waited for a list of information. Instead, we immediately got errors. The volume label could not be read. After searching online, we discovered we could run commands to move the tape around and try to read some tape labels, but then we got I/O errors.

Fortunately, the customer had made two copies of the backup tapes, so I picked up the second tape. At that point, I was told that they'd used an unusual block size. That turned out to be the key. We tried 0, 512, and 1024, but hadn't thought about 262144. Once we changed the block size, we were able to restore the data and all went as expected.

With the data restored, we copied it to a Windows machine to burn a DVD copy of the data. The customer then loaded the DVD onto their own Windows machine, and sent it to their server via FTP.

Some takeaways from this project: If you're archiving data, either keep the hardware and software that's capable of reading it, or periodically transfer important old data to newer media. And if you use a different block size when writing data, be sure that's documented—preferably on the tape itself.

I enjoy solving problems, and this was no exception. Now if I'm asked to restore an old Sysback tape, I know I can do it—provided Jeff and Pete are again willing to lend me a hand.

ASO the First Phase in Autonomic Tuning

I've touched on Active System Optimizer (ASO) before, but now that Nigel Griffiths has released an ASO video, it seems an appropriate time to expound on this topic.

To run ASO, you must be at AIX7.1 TL01 or greater on POWER7 hardware running in POWER7 mode. ASO is installed by default on AIX 7. It's not supported on older AIX releases or older hardware. (It will appear to be running, but will actually be hibernating.)

Nigel recommends running:

* oslevel –s to verify your AIX version

* lslpp –L | grep –I optimi to help verify that the ASO fileset is installed

* lsconf | grep ^Processor to verify that your LPAR is running in POWER7 mode.

ASO works under the covers; you don't need to do anything to start it. It optimizes workloads based on AIX kernel statistics and the POWER7 hardware performance counters. It's designed to improve cache and memory affinity by dynamically evaluating tuning options and making changes -- including moving processes and memory -- on the fly. It conducts pre- and post-monitoring to ensure that the changes improve performance. If improvement isn't detected, ASO backs out the changes, hibernates and tries again later. Listen to Nigel's presentation for details on this.

ASO provides cache affinity, aggressive cache affinity and memory affinity. It monitors performance and detects situations where threads can be moved from one chip to another to utilize the closer L3 cache.

ASO operates best on multi-threaded workloads. The jobs it monitors should be stable and long-running so it can most effectively make changes to the workload. It also needs to be a busy LPAR, otherwise you don't gain much by trying to move things around. By "busy," I mean that the processes need at least a 10-second lifetime. If applications come and go more quickly, ASO cannot make recommendations. If you've done manual tuning, either on your own or based on recommendations from IBM Support, ASO will simply hibernate rather than monitor and override your changes. In addition, specific processes can be marked as "don't bother these," and ASO won't impact them.

ASO runs as an SRC kernel service. To locate it, search your machine for the aso process.

Use these commands to get ASO running:

1) start the kernel service with startsrc –s aso

2) run asoo –o aso_active=1

According to Nigel, the aso process uses very minimal CPU time, so this shouldn't add much additional overhead on the system.

Logs are located in the /var/log/aso/* directory. You'll see two files:

* aso.log, which has on/off/hibernating information

* aso_process.log, which provides details of actions and modified processes

Nigel says this log file isn't formally documented, but you should be able figure out what it's doing.

The man page says ASO can run outside of SRC, but this should probably only be done for debugging. You can also set shell variables before starting processes and apps. This provides some control over how they function with ASO.

As always, Nigel has much more than I can cover here. For instance, he shows real life examples of ASO running on his machine, output from the logfile, and more.

To wrap up, he tells us that ASO is largely set and forget. ASO uses near zero CPU when running, and it gently applies changes, tests behavior and undoes the changes if necessary. Because ASO is good for complex, multithreaded, long-running applications, it can move things around inside of your LPAR if they're spread across CPUs.

Best of all, this is, as he says, just the first phase of clever autonomic affinity tuning. So keep your eyes open for what's ahead.

A Good Look at PowerHA

Another great Virtual User Group webinar recently took place, this one featuring Shawn Bodily's presentation, "Introduction to PowerHA SystemMirror for AIX Standard Edition." Be sure to get the presentation materials and listen to the replay. And look forward to the next VUG webinar, when this topic will be continued.

This material is very similar to the presentation on the same topic at last fall's IBM Technical University. That's an indication of the interest surrounding this solution. As Shawn notes in his presentation, IBM has unveiled 23 major releases of the product, an average one release per year. More than 12,000 customers use it worldwide. While Standard Edition is the focus of Shawn's presentation, he points out that PowerHA SystemMirror for AIX Enterprise edition allows for multi-site cluster management, while also including all the functionality of Standard edition.

Shawn notes that "PowerHA gives you cluster management for the data center. The software monitors, detects and reacts to events. It establishes a heartbeat between the systems, and enables automatic switch-over." He then defines what HA solutions generally try to do, which is eliminate single points of failure. The goal is to reduce downtime, but HA can also help you with planned downtime as well. It might not be fault tolerant, but it will be fault resistant.

One chart shows how failure points can be eliminated. To eliminate node failure, use multiple nodes. To eliminate VIO failover, use dual VIO servers. To eliminate site failure, deploy additional sites. Other items are covered, but again, the idea is to build in redundancy so you can continue to provide access to the applications and data that the business runs on.

Shawn discussed his customer interactions (an area I've touched on previously):

"First, another IBM representative will tell the customer about the hardware and the systems' reliability, availability and serviceability (RAS) features. Then a second rep will discuss live partition mobility and how it seamlessly shifts logical partitions from one frame to another. So after 20-30 minutes of hearing about how the hardware never fails, THEN Shawn must step in and explain why the customer should be concerned with high availability and disaster recovery. That's one tough act to follow."

Some additional slides introduce Live Partition Mobility and explain how it allows you to move your live running operating system and application from one physical frame to another. Of course, that's a hardware maintenance solution. What about software maintenance? With PowerHA, you can fail over to the other node, upgrade your application or your operating system, then fail back and do the same thing on the other side. Shawn notes that basically, PowerHA performs the functions that LPM doesn't. He also gets into how PowerHA is used to recover from node failures, network failures, loss of shared storage access, and -- with version 7.1 based on Cluster Aware AIX technology -- rootvg errors. Finally, Shawn covers the differences between PowerHA 6.1 and 7.1.

Shawn makes these additional points:

* Remember, we can fail over from one system to one system, from one system to any other system, from any system to one, and from any to any. We can also fail over between different versions of hardware from the same or different families, assuming you can live with the performance degradation, and after you verify that the version of the operating system you want to run will work with your particular configuration. Failing POWER7 to POWER6 or POWER5 could conceivably work as long as you verify that your particular setup is supported.

* Often you'll want to make your service IP address highly available, along with your application server and your shared storage.

* You can create user defined resources, custom resource groups and more granular resource group options. You can set up resource group dependencies. In his example, you might want to be sure your database is running before you try to start application servers, so you could configure that as a dependency.

* You can configure different priorities and choose which nodes to your resource starts on. This provides a great deal of flexibility and control when setting things up.

* You can also configure things to automatically run DLPAR and COD operations. So you could have a very "skinny" standby node, but when needed it could perform operations on the HMC and bring additional memory and CPU resources online.

* You can have application monitors so you can take actions if PowerHA detects that the application has gone down, and you can set up file collections to have the software help you keep config files or other important files kept in sync across the cluster. This is meant to support all regular files, as opposed to things like trying to keep password files in sync.

* Configuration and smart assistants are available to configure clusters. System Director plugins are also available, both for managing and monitoring your cluster's state.

* The CAA command -- /usr/sbin/clcmd -- can be used to distribute commands across all cluster nodes.

* A cluster test tool can be used to validate clusters. This is also a good way to run tests across many different clusters in the environment, to ensure that we're running the same tests across all of the machines.

Shawn's final slide lists these great resources:

* IBM Redbook: PowerHA SystemMirror 7.1 for AIX

* PowerHA & Aix Support & Compatibility Matrix

* PowerHA Hardware Support Matrix

Incidentally, you can follow Shawn on Twitter.

I really enjoy these webinars, and of course the availability of the replays and presentation materials is a huge convenience. Hopefully my writing about VUG webinars encourages you to take the time and listen for yourself.

Note: Completely unrelatedly, I'm a YouTube sensation. Well, maybe not, but I'm there. Five new videos just went live where you can see me and some other IBM Power Champions talking with IBM's Ian Jarman at last fall's Technical University.

 

 

 

More on Shared Storage Pools

Back in 2010 I wrote about the changes that were coming to VIOS. One of those big changes, shared storage pools, is now a reality. This gives admins another option to consider when setting up disks on Power servers.

In larger companies, disk changes are typically implemented by SAN teams with many other responsibilities and, often, different priorities. However, by allocating storage to the servers up front and setting it up in a storage pool, admins can manage shared storage pools. In doing so, we can be more responsive to requirement changes. And with thin provisioning, we can determine the amount disk we actually use on each server. For the first time since the days of internal disks and expansion drawers, disk is back under our control.

Here's how Nigel Griffiths explains shared storage pools:

"The basic idea behind this technology... is that [VIO servers] across machines can be clustered together and allocate disk blocks from large LUNs assigned to all of them rather than having to do this at the SAN storage level. This uses the vSCSI interface rather than the pass through NPIV method. It also reduces SAN admin required for Live Partition Mobility -- you get the LUN available on all the VIOS and they organise access from there on. It also makes cloning LPARs, disk snapshots and rapid provisioning possible. Plus thin provisioning -- i.e., disk blocks -- are added as and when required, thus saving lots of disk space."

Continuing from last week, here's more from Nigel's presentation.

Since shared storage pools are built on top of cluster-aware AIX, the lscluster command also provides more information, including: lscluster –c  (configuration), lscluster –d  (list all hdisks), lscluster –i  (network interfaces), lscluster –s  (network stats).

In the demo, he also discusses adding disk space and assigning it to client VMs. Keep in mind that you cannot remove a LUN from the pool. You can replace a LUN but you can't remove one.

He also covers thin and thick provisioning using shared storage pools and shows you how to conduct monitoring. Run topas on your VIOS and then enter D (make sure it's upper-case) so you can watch the disk I/O get spread across your disks in 64 MB chunks. From there, Nigel covers how to set up alerts on your disk pool. If you're using thin provisioning, you must ensure you don't run out of space.

Nigel also shares his script, called lspool. It's designed to do the work of multiple scripts by presenting all of the critical information at one time instead of running multiple commands:

# lspool list each cluster and for each list its pools and pool details
~/.profile
clusters=`cluster -list | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo "Cluster list: " $clusters
for clust in $clusters
do
pools=`lssp -clustername $clust | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo Pools in $clust are: $pools
for pool in $pools
do
lssp -clustername $clust | sed '1d' | grep $pool | read p size free
totalLU numLUs junk
let freepc=100*$free/$size
let used=$size-$free
let usedpc=100*$used/$size
echo $pool Pool-Size: $size MB
echo $pool Pool-Free: $free MB Percent Free $freepc
echo $pool Pool-Used: $used MB Percent Used $usedpc
echo $pool Allocated: $totalLU MB for $numLUs Logical Units
alert -list -clustername $clust -spname $pool | sed '1d' | grep $pool
| read p poolid percent
echo $pool Alert-Percent: $percent
if [[ $totalLU > $size ]]
then
let over=$totalLU-$size
echo $pool OverCommitted: yes by $over MB
else
echo $pool OverCommitted: no
fi
done
done

Nigel examines snapshots and cloning with shared storage pools, noting that the different commands -- snapshot –create, snapshot –delete, snapshot –rollback and snapshot –list -- use different syntax. Sometimes it asks for a –spname flag, other times it asks for a –sp flag. Pay attention so you know the flags that are needed with the commands you're running. He also demonstrates how some of this management can be handled using the HMC GUI.

The viosbr command is also covered. I discussed it here.

Nigel recommends that you get started by asking the SAN team to hand over a few TB that you can use for testing. Also make sure your POWER6 and POWER7 servers are at the latest VIOS 2.2 level. It's worth the effort. This technology will save time, boost efficiency and increase your overall responsiveness to users.

Finally, here's Nigel's shared storage pools cheat sheet:

1. chdev -dev <device name> -attr reserve_policy=no_reserve
2. cluster -create -clustername galaxy -repopvs hdisk2
-spname atlantic -sppvs hdisk3 hdisk5 -hostname bluevios1.ibm.com
3. cluster –list
4. cluster -status -clustername galaxy
5. cluster –addnode –clustername galaxy –hostname redvios1.ibm.com
6. cluster -rmnode [-f] -clustername galaxy -hostname redvios1.ibm.com
7. cluster –delete –clustername galaxy
8. lscluster –s or –d or –c or –i = CAA command
9. chsp –add –clustername galaxy -sp atlantic hdisk8 hdisk9
10. chsp -replace -clustername galaxy -sp atlantic -oldpv hdisk4 -newpv hdisk24
11. mkbdsp -clustername galaxy -sp atlantic 16G
-bd vdisk_red6a -vadapter vhost2 [-thick]
12. rmbdsp -clustername galaxy -sp atlantic -bd vdisk_red6a
13. lssp -clustername galaxy -sp atlantic -bd
14. lssp -clustername galaxy
15. alert -set -clustername galaxy –spname atlantic -value 80
16. alert -list -clustername galaxy -spname atlantic
17. errlog –ls
18. snapshot -create name -clustername galaxy -spname atlantic -lu LUs
19. snapshot -delete name -clustername galaxy -spname atlantic -lu LUs
20. snapshot -rollback name -clustername galaxy -spname atlantic -lu LUs
21. snapshot –list -clustername galaxy -spname atlantic
22. viosbr -backup -clustername galaxy -file Daily -frequency daily -numfiles 10
23. viosbr -view -file File -clustername Name …
24. viosbr -restore -clustername Name …
25. lsmap -clustername galaxy –all

Take the time to listen to the replay, and you'll learn even more. I highly recommend it.

Getting Started with Shared Storage Pools

The December AIX Virtual User Group webinar featured Nigel Griffiths' discussion of phase 2 of shared storage pools. If you didn't tune in, download the presentation materials and listen to the replay.

The new shared storage pool functionality is enabled with the latest PowerVM 2.2 service pack, and is a feature of PowerVM Standard and Enterprise. If you already have PowerVM, simply download the VIO server fixpack to obtain these new features. (Note: Because this TL is based on AIX 6.1 TL7, your NIM server must be at AIX 6.1 TL7 or AIX 7.1 TL1 to use your NIM server with your VIO server.)

 One thing to note, as Nigel points out in the presentation, is that the most common VIOS storage options have been around for some time:

1) Logical volumes, created from a volume group and presented to client LPARs
2) Whole local disks
3) SAN LUNs
4) File-backed storage, either from a file system on local disk or a file system on SAN disks
5) NPIV LUNs from SAN

Nigel then discusses the newest option: using SAN LUN disks that are placed into a shared storage pool. This new option, he emphasizes, doesn't eliminate any of the other options. It does not portend the death of NPIV. It's just an additional VIOS storage choice we now have.

Listen to the replay or look over the slides to gather Nigel's thoughts on the benefits of shared storage pools. He explains that fibre channel LUNs and NPIV can be complex. They require knowledge of the SAN switch and the SAN disk subsystem. If you need to make changes, it might take your SAN guys awhile to implement them. This can slow overall responsiveness. That's to say nothing of smaller organizations that don't have dedicated SAN guys. Live partition mobility can be tough work if your disks aren't pre-zoned to the different frames.

With a shared storage pool you pre allocate the disk to the VIO servers. Then it's under your control. You can more easily allocate the space to your virtual machines.

POWER6 and POWER7 servers (including blades) are needed to use shared storage pools. At minimum you should allocate a 1 GB of LUN for your repository and another 1 GB of LUN for data, but in order to be useful, in most cases you'll need much larger LUN(s) – think terabytes of disk -- if you plan to do much with it. 

Your VIOS must have the hostname set correctly to resolve the other hostnames. In Nigel's experience he couldn't use short hostnames -- they had to be changed to their fully qualified names.

He also recommends giving your VIOS a CPU and at least 4 GB of memory. "Skinny" VIOS servers aren't advisable with shared storage pools. Currently, the maximum number of nodes is four, the maximum physical disks in a pool is 256, the maximum virtual disks in a cluster is 1,024, and the number of clients is 40. A pool can have 5 GB to 4 TB of individual disks, and storage pool totals can range from 20 GB to 128 TB. Virtual disk capacity (LU) can be from 1 GB to 4 TB, with only one repository disk.

If you played around with phase one, you'll find that many of your limits have been removed. Now you can use shared storage pools for live partition mobility, perform non-disruptive cluster upgrades and use third party multi-pathing software.

You cannot have active memory sharing paging disks on shared storage pool disks.

Nigel covers the relevant terminology (clusters, pools, logical units, etc.).  He also demonstrates how to actually prepare and set up your disks. In a nutshell you must get your LUNs online and zoned to your VIO servers, and you need to set your reserve policy to no_reserve on your LUNs.

After covering the commands for managing clusters -- cluster –create, cluster –list, cluster –status, cluster –addnode, cluster –rmnode and cluster –delete -- he recommends creating the cluster on one of the VIO servers and then adding additional VIO servers to the shared storage pool cluster. From there, you can allocate space to the VM clients.

Next week I'll have more information from Nigel’s presentation, including scripts and cheat sheets. In the meantime, why not upgrade your test machine's VIO servers to the latest level so you can try this functionality?

Verifying Microcode Levels

As great as POWER7 servers are, plenty of older machines still run AIX. And as great as that is, even better is that, through the use of certain tools, you can easily verify that these older machines are running the latest versions of system firmware and microcode.

Recently a "Top Gun" CE reminded me about the Microcode Discovery Service. It's a handy tool that allows you to see if your microcode is up to date.

From IBM Support:

"Microcode Discovery Service [MDS] is used to determine if microcode installed on your IBM System p or RS/6000 systems is at the latest level.

"MDS relies on an AIX utility called Inventory Scout. Inventory Scout is installed by default on all AIX 5 systems, and also on some later levels of AIX 4.3."

As noted by IBM, there are three ways to run MDS:

• Run a signed Java applet that connects to Inventory Scout daemon processes on hosts to be surveyed.
• Run Inventory Scout either manually or by script, and upload the resulting survey files to the MDS website for analysis. Refer to the User's Guide for instructions on running Inventory Scout.
• The MDS Microcode CD-ROM is recommended for systems that are not internet connected. An image of this CD-ROM is available online, or you can order a physical disk. For more information or to download a CD-ROM image of this tool, visit: MDS Microcode CD-ROM.

Under the heading, "Preparing to use the Microcode Directory Service," there's a description of the Inventory Scout:

"Inventory Scout is a utility that runs on System p hosts. For AIX version 5 and later it is part of the standard install. In case it is not installed, refer to the User's Guide for instructions. ...

"The MDS applet is capable of performing surveys of more than one host at a time, and creating a combined microcode report. If using the MDS applet, then the following additional conditions must be met.

• "Each host to be surveyed must be running Inventory Scout in a daemon process. These daemon processes are not started by default. Refer to the User's Guide for instructions on how to configure Inventory Scout to run as a daemon.
• "Java support must be enabled in your browser. To enable this support, see the Preferences or Tools options on your browser.
• "Your company must allow applets to establish TCP connections to the hosts to be surveyed."

I clicked on the applet option

 

and allowed it to run:
 


This screen allows you to add a host and give it an IP address, password and (optionally) the port if you've changed it.
 



If the wrong IP address is entered, you'll get this screen.


 

I ran passwd on the invscout userid so that I knew the password for the invscout account. Then I ran invscoutd –d100000 on the command line of the machine I was going to scan so I could activate the daemon and get it to listen for the applet to connect. Originally invscoutd was listening for 50,000 bytes. It needs to listen for 100,000 bytes, which is why the –d flag is required.

Once I successfully added the host, I clicked start:




That brought up this report.



It found three devices and displayed the system firmware that needed to be updated. I clicked the link and copied the files to the machine that was running the applet. Then I moved those files over to the Power system that needed to be updated.

In my case I created /tmp/mcode, and copied the files to that directory. Then I ran rpm –ivh -–ignoreos *.rpm to get the information and microcode loaded into the /etc/microcode directory.

Next, I ran diag, task selection:

On this screen, I selected the "Microcode Tasks" option:

Then I selected "Download Latest Available Microcode" and hit enter:

Since I ran the rpm command earlier, I knew the files should be in /etc/microcode. I selected that option:

Then I selected "All Resources" and then hit the F7 key to let the microcode install:

 


I actually updated the system firmware via the HMC, though I could have used the command line if my system wasn't managed by an HMC. Once my updates were complete, I reran the MDS applet. No further updates were needed:

 

Of course there are other methods for verifying that your microcode is up to date. How do you prefer to do this in your environment?

What to Say About What You Do

This recent Anthony English post got me thinking. When someone asked him what he did, he wasn't sure how to respond. How do you answer that question? Can you explain what you do in a nice 30-second elevator pitch?

Luckily for me, Watson recently made a repeat appearance on "Jeopardy!" Watson has become the basis for my "pitch": I simply tell people I work on the same servers that they saw on TV with Alex Trebek.

Of course, I'll never be able to count on "my" servers appearing on television with any regularity, so I still need other ways to explain to others how I make my living. Or do I? As I've noted previously, telling people you work on computers can have some unwanted consequences. In their minds my admission is their opening to ask me to come over and resurrect their old, possibly virus-ridden machines. ("Can’t you just add a hard drive or memory or something?") I generally counter those requests by explaining that I work on large enterprise servers running enterprise operating systems--in other words, small machines aren't my specialty. Of course to a lot of folks, a computer is a computer.

(As an aside, why is it considered bad form to ask your doctor and lawyer acquaintances for free legal or medical advice, but few think twice about asking computer nerds they know for free help? Maybe they figure we have nothing better to do. Maybe we need to start quoting our hourly rates.)

So what do you say when you're asked what you do? Do you talk about a typical day at work (one where machines haven't blown up)? I've heard a system administrator liken his job to a plumber's: nobody notices or needs one until something stinks. (That's figuratively, one can only hope, in the admin's case.)

If you're reading this blog, I figure you're involved with provisioning machines. You may not have built them, but you do care for them day to day. Interacting with your machines as much as you do, over time you may even get close to them. However, eventually you must put them out of their misery and upgrade to newer gear. (Another aside: It's actually funny how quickly this cycle can run. You just installed the fastest, shiniest new hardware, but in a few short years you're yearning for that new, state-of-the-art box.)

Ultimately, the way I explain what I do depends entirely on my audience and their frame of reference. To those in the industry, it's simple: I start by saying I sell and install IBM's Power hardware line and specialize in AIX. But to non-industry folks, I can't have their eyes glazing over from my tales of patching, upgrading, installing, cabling, provisioning and deploying machines. Not to mention backups, restores, clones, LUNs, mirrors, migrations, copies, archives, scripts, cron jobs and the like. Maybe it's enough for them to know that my specialty is enterprise IBM hardware. Or maybe I just say I work with computers.

What do you do?

Protecting Your Data with mirrorios

In the "good old days" of AIX administration, companies had standalone servers, and rootvg lived on internal disks. We always had at least one pair of internal disks, mirroring them to one another. In the event of a disk failure, you'd unmirror the disks and then replace the failing/failed disk. This was usually accomplished on the fly with hot swap disks. Typically the end users never even knew there was a problem.

Those who still run servers on physical internal disks still need to make sure that they're mirrored in the event a disk needs to be replaced. But even with companies that have moved on to virtualization technology, much of this thinking is preserved today with dual VIO servers and virtual SCSI devices. If you lose one physical path to the storage, the VIO client uses the path provided by the redundant VIOS to keep running.

Most customers I work with these days boot their LPARs (I know, I'm supposed to call them virtual servers, but old habits are hard to break) from SAN. The disk protection and physical disk replacement happens on the back end with a SAN team. Although I do see customers where the SAN guy and the AIX admin are the same person, in all cases the data protection occurs behind the scenes as far as AIX is concerned. The hdisk isn't affected as far as the OS knows.

When installing your VIOS and booting it from internal disks, it's still a good idea to mirror that disk to the other internal disk that's assigned to the same bus on the VIOS. With split backplanes and dual VIO servers, this thinking just needs to be taken a step further: be sure to mirror all of the disks on all of your VIO servers, assuming they're booting locally.

To help in this regard, VIOS has a built-in command called called mirrorios. When you run it, you'll be prompted to reboot your machine when the mirror operation completes. However, that can be deferred by simply running the command, mirrorios –defer.

When this command completes, check your bootlist. You'll find that it hasn't been updated with the disk that you just mirrored to. To remedy this, you must manually run a bosboot on the new disk you mirrored to, and then update your bootlist to reflect the change. If you're wondering why mirrorios can't perform both steps, you're not alone. This is supposed to be an appliance, after all.

So what other methods do you use to protect your data? Surely you take mksysb images and backup your system using TSM or some other method. This of course, brings up the familar but important question: Have you tested your restore procedures lately? Are you sure they work?

Finally, have you made sure your VIO servers are set up correctly? As with a high availability cluster, the wrong time to find out that things aren't set up correctly is when you really need them to work.

Connecting to a Remote HMC

What do the best practices documents tell us about HMC private networks when communicating between HMC and flexible service processor (FSPs)? Is a private network switch or VLAN really needed between the HMC and the FSP? Can an HMC in a remote data center be used to manage machines over regular network links?

This 2007 document, authored by IBMers Ron Barker, Minh Nguyen, and Shamsundar Ashok, states:

"The network connection between the HMC and the FSP can be either private or open on low-end to mid-range servers. Private is preferred, and therefore a best practice. A private network is required for systems that have a BPA, such as the models 590, 595 and 575.

"In an open configuration, the FSP’s IP addresses must be set manually on each managed server. They cannot be DHCP clients of any server other than a managing HMC.

"Addresses can be set using the Advanced System Management Interface (ASMI) on the FSP. This involves directly connecting a laptop to one of the ports on the FSP and using HTTPS to log into one of the two pre-defined IP addresses. The HMC1 port defaults to 192.168.2.147; HMC2 defaults to 192.168.3.147. The systems administrator can login as user 'admin' using the default password 'admin,' which should be changed during the initial installation for security reasons. If no laptop is available, an ASCII terminal can be used on the native serial port to access the FSP menus in character mode.

"Remember, with POWER7, the addresses have changed to:

Service processor A HMC1 169.254.2.147,  HMC2 169.254.3.147 and
Service processor B (if installed) HMC1 169.254.2.146  HMC2 169.254.3.146

"Open networks are used for communications between a logical partition and the HMC. This connection is largely to facilitate traffic over the Resource Monitoring and Control (RMC) subsystem, which is the backbone of Service Focal Point (SFP) and required for dynamic resource allocation. The open network also is the means by which remote workstations may access the HMC, and it could be the path by which an HMC communicates with IBM Service through an Internet connection."

Though HMCs will be going away in favor of SDMC, the transition will be gradual. For the time being we need to keep our HMC skills sharp, and this is one question that frequently arises when customers plan to add new systems to an environment.

Recently I helped a client set up HMC communications over an open network in just this manner. We found that our FSP ports were still going 100MB, as the ports were requested to be set to 1000/Full Duplex. They wouldn't link up to the network at that speed.

Once we had the HMC on the open network, we pointed the HMC to the new IP addresses we'd configured on the FSPs on the local machines. That worked. We then added managed systems that were located in a remote data center without a problem. Finally, we did the same thing with an HMC from the remote data center to manage the machines in the local data center.

While I wouldn't recommend using a remote HMC for day-to-day tasks if a local one is available to you, this is a viable option when setting up machines.

How is the HMC set up in your environment? Please share your experiences in Comments.

An AIX Migration Tip Leads the Grab Bag

It's been awhile since I've given you a grab bag of links and tips. I'll start with a personal experience.

A recent client with an Power server running an Oracle database was migrating from AIX 5.3 to AIX 6.1 Certain settings they specified in their /etc/tunables/nextboot took effect when they booted AIX6, and they couldn't figure out why Oracle was running so horribly. Jobs that normally ran for a few minutes were taking nearly an hour to process. Luckily, someone noticed some messages regarding changes to restricted tunables. Upon checking /etc/tunables/lastboot.log, they saw:

Setting maxperm% to 30
Warning: a restricted tunable has been modified
Setting maxclient% to 30
Warning: a restricted tunable has been modified
Setting strict_maxperm to 1
Warning: a restricted tunable has been modified

Once they changed /etc/tunables/nextboot and rebooted, Oracle ran like a champ and the machine was fine. So add this to your migration checklist: Try the AIX 6.1 default settings first, then make modifications if needed post-upgrade. And be sure to check the tunable settings that are carried over with a migration.

You know that I frequently link to IBMer Nigel Griffiths-–follow him on Twitter as @mr_nmon. In addition to sharing some hints and tips about IBM Systems Director, he's posted a number of entries covering options for monitoring entire physical systems (as opposed to monitoring on a virtual server or a VM by VM basis).

For instance, here are 22 things you should do before setting up Systems Director. And here are eight things you should do once you're running it. The latter installment tells you, among other things, how your boss can run Director from an iPad.

Nigel also offers systems monitoring tips involving Director, lpar2rrd, topas CEC analyzer, IBM Tivoli Monitoring and Ganglia.

For more on Systems Director, IBM has these YouTube videos featuring user experience designer Greg
Hintermeister:

• Monitor and repair your system using IBM Systems Director
• Deploy and optimize your systems
• Plan and document changes in your data center

Recent AIX Virtual User Group meetings have also included Systems Director info. Listen to the replays, download the presentation materials and sign up for future meetings here.

As I've said before, you'll find a lot of AIX knowledge on Twitter. @nicolettemcf, @cgibbo, @ibmaix, @aixmag, @aixdownunder are just a few of the users I follow. Search on #AIX and I'm sure you'll find others you like.