Script Changes

I received an interesting e-mail from a mailing list. Included was this information submitted by Dean Rowswell:

1. Turn on PuTTY logging
2. Copy and paste these 6 commands first:

lshmc -v
lshmc -V
lshmc -r
lshmc -n
lshmc -b
lssysconn -r all
lssyscfg -r sys
lssyscfg -r frame
lshmc -n -F clients
cat /opt/hsc/data/.hmc/.removed
lspartition -dlpar
lspartition -sfp

3. Copy and paste these commands last:

for MANAGEDSYS in `lssyscfg -r sys -F type_model*serial_num`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
echo " ============MANAGED SYSTEM --> LIC level"
lslic -m ${MANAGEDSYS} -t sys
echo " ============MANAGED SYSTEM --> processor config"
lshwres -m ${MANAGEDSYS} -r proc --level sys
echo " ============MANAGED SYSTEM --> lpar processor usage"
lshwres -m ${MANAGEDSYS} -r proc --level lpar -F
lpar_name:curr_proc_mode:curr_sharing_mode:run_proc_units:run_procs
echo " ============MANAGED SYSTEM --> memory config"
lshwres -m ${MANAGEDSYS} -r mem --level sys
echo " ============MANAGED SYSTEM --> lpar memory usage"
lshwres -m ${MANAGEDSYS} -r mem --level lpar -F lpar_name:run_mem
echo " ============MANAGED SYSTEM --> lpar status"
lssyscfg -m ${MANAGEDSYS} -r lpar -F name:state
echo " ============MANAGED SYSTEM --> cuod processor config"
lscod -m ${MANAGEDSYS} -t cap -r proc -c cuod
echo " ============MANAGED SYSTEM --> cuod memory config"
lscod -m ${MANAGEDSYS} -t cap -r mem -c cuod
echo " ============MANAGED SYSTEM --> drawer config"
lshwres -m ${MANAGEDSYS} -r io --rsubtype unit
echo " ============MANAGED SYSTEM --> bus config"
lshwres -m ${MANAGEDSYS} -r io --rsubtype bus
echo " ============MANAGED SYSTEM --> slot config"
lshwres -m ${MANAGEDSYS} -r io --rsubtype slot
echo " ============MANAGED SYSTEM --> slot config summary"
lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
drc_name:description:lpar_name:lpar_id
echo " ============MANAGED SYSTEM --> virtual ethernet"
lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype eth --level sys
echo " ============MANAGED SYSTEM --> virtual ethernet all lpar"
lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype eth --level lpar
echo " ============MANAGED SYSTEM --> virtual scsi all lpar"
lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype scsi --level lpar
for LPAR in `lssyscfg -r lpar -m ${MANAGEDSYS} -F name`
do
echo " ============LPAR --> ${LPAR} --> CPU resources"
lshwres -r proc -m ${MANAGEDSYS} --level lpar --filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> Memory resources"
lshwres -r mem -m ${MANAGEDSYS} --level lpar --filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> Physical adapters"
lshwres -r io --rsubtype slot -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> Virtual Ethernet config"
lshwres -r virtualio --rsubtype eth --level lpar -m ${MANAGEDSYS}
--filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> Virtual SCSI config"
lshwres -r virtualio --rsubtype scsi --level lpar -m ${MANAGEDSYS}
--filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> LPAR config"
lssyscfg -r lpar -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo " ============LPAR --> ${LPAR} --> LPAR profiles"
lssyscfg -r prof -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
done
done

4. Copy and paste these lines to add the information to my LPAR resource allocation spreadsheet
NOTE: when I do a paste into Excel click on the paste options and select "Text import wizard"

for MANAGEDSYS in `lssyscfg -r sys -F type_model*serial_num`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
unit_phys_loc:bus_id:phys_loc:description
done

For System p machines that already have the physical resources assigned:

for MANAGEDSYS in `lssyscfg -r sys -F type_model*serial_num`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
unit_phys_loc:bus_id:phys_loc:description:lpar_id:lpar_name
done

5. To capture any errors and events

lssvcevents -t hardware
lssvcevents -t console

After messing around with the script, I wanted to get it working from cron. I'm unable to run scripts on my HMC, but after looking here, I set up my ssh keys so I could auto login from my AIX machine to my HMC.

Then I modified the above script so I could run it from my AIX machine and have it connect to my HMC using ssh. Now I can run the job out of cron on my AIX machine instead of messing with putty.

#!/usr/bin/ksh
#
# scriptname -m hmchostname -l hmcuser
#
#
hmc=hmc.ip.address

user=hscroot

while getopts m:l: option
do
case $option in
m) hmc="$OPTARG";;
l) user="$OPTARG";;
esac
done

echo "HMC Information:"
echo ""
ssh $hmc -l $user 'date'
ssh $hmc -l $user 'lshmc -v'
ssh $hmc -l $user 'lshmc -V'
ssh $hmc -l $user 'lshmc -r'
ssh $hmc -l $user 'lshmc -n'
ssh $hmc -l $user 'lshmc -b'
ssh $hmc -l $user 'lssysconn -r all'
ssh $hmc -l $user 'lssyscfg -r sys'
ssh $hmc -l $user 'lssyscfg -r frame'
ssh $hmc -l $user 'lshmc -n -F clients'
ssh $hmc -l $user 'cat /opt/hsc/data/.hmc/.removed'
ssh $hmc -l $user 'lspartition -dlpar'
ssh $hmc -l $user 'lspartition -sfp'

for MANAGEDSYS in `ssh $hmc -l $user "lssyscfg -r sys -F type_model*serial_num"`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"

echo " ============MANAGED SYSTEM --> LIC level"
ssh $hmc -l $user "lslic -m ${MANAGEDSYS} -t sys"
echo " ============MANAGED SYSTEM --> processor config"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r proc --level sys"
echo " ============MANAGED SYSTEM --> lpar processor usage"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r proc --level lpar -F
lpar_name:curr_proc_mode:curr_sharing_mode:run_proc_units:run_procs"
echo " ============MANAGED SYSTEM --> memory config"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r mem --level sys"
echo " ============MANAGED SYSTEM --> lpar memory usage"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r mem --level lpar -F
lpar_name:run_mem"
echo " ============MANAGED SYSTEM --> lpar status"
ssh $hmc -l $user "lssyscfg -m ${MANAGEDSYS} -r lpar -F name:state"
echo " ============MANAGED SYSTEM --> cuod processor config"
ssh $hmc -l $user "lscod -m ${MANAGEDSYS} -t cap -r proc -c cuod"
echo " ============MANAGED SYSTEM --> cuod memory config"
ssh $hmc -l $user "lscod -m ${MANAGEDSYS} -t cap -r mem -c cuod"
echo " ============MANAGED SYSTEM --> drawer config"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype unit"
echo " ============MANAGED SYSTEM --> bus config"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype bus"
echo " ============MANAGED SYSTEM --> slot config"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype slot"
echo " ============MANAGED SYSTEM --> slot config summary"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
drc_name:description:lpar_name:lpar_id"
echo " ============MANAGED SYSTEM --> virtual ethernet"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype
eth --level sys"
echo " ============MANAGED SYSTEM --> virtual ethernet all lpar"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype
eth --level lpar"
echo " ============MANAGED SYSTEM --> virtual scsi all lpar"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r virtualio --rsubtype
scsi --level lpar"

for LPAR in `ssh $hmc -l $user "lssyscfg -r lpar -m ${MANAGEDSYS} -F name"`
do
echo " ============LPAR --> ${LPAR} --> CPU resources"
ssh $hmc -l $user "lshwres -r proc -m ${MANAGEDSYS} --level lpar
--filter lpar_names=${LPAR}"
echo " ============LPAR --> ${LPAR} --> Memory resources"
ssh $hmc -l $user "lshwres -r mem -m ${MANAGEDSYS} --level lpar
--filter lpar_names=${LPAR}"
echo " ============LPAR --> ${LPAR} --> Physical adapters"
ssh $hmc -l $user "lshwres -r io --rsubtype slot -m ${MANAGEDSYS}
--filter lpar_names=${LPAR}"

echo " ============LPAR --> ${LPAR} --> Virtual Ethernet config"
ssh $hmc -l $user "lshwres -r virtualio --rsubtype eth --level lpar -m
${MANAGEDSYS} --filter lpar_names=${LPAR}"
echo " ============LPAR --> ${LPAR} --> Virtual SCSI config"
ssh $hmc -l $user "lshwres -r virtualio --rsubtype scsi --level lpar
-m ${MANAGEDSYS} --filter lpar_names=${LPAR}"
echo " ============LPAR --> ${LPAR} --> LPAR config"
ssh $hmc -l $user "lssyscfg -r lpar -m ${MANAGEDSYS} --filter
lpar_names=${LPAR}"
echo " ============LPAR --> ${LPAR} --> LPAR profiles"
ssh $hmc -l $user "lssyscfg -r prof -m ${MANAGEDSYS} --filter
lpar_names=${LPAR}"
done
done

for MANAGEDSYS in `ssh $hmc -l $user "lssyscfg -r sys -F type_model*serial_num"`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
unit_phys_loc:bus_id:phys_loc:description"
done

for MANAGEDSYS in `ssh $hmc -l $user "lssyscfg -r sys -F type_model*serial_num"`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
ssh $hmc -l $user "lshwres -m ${MANAGEDSYS} -r io --rsubtype slot -F
unit_phys_loc:bus_id:phys_loc:description:lpar_id:lpar_name"
done

ssh $hmc -l $user 'lssvcevents -t hardware'
ssh $hmc -l $user 'lssvcevents -t console'

Feel free to improve upon what Dean and I have done so far. I'll add your contribution to a future posting.

IBM Power Announcement: Not Just Another Renaming

I spent the first part of my career working on OS/400, but since then I have been much more focused on the System p and AIX. Like a lot of you, I've been around long enough to recall when these machines were known as the AS/400 and RS/6000. I was amused when a coworker forwarded me this article on April Fool’s Day. It talks about IBM and the name changes they have made over the years to the AS/400.

Ironically, the day after that article was published, April 2, 2008, IBM announced yet another name change. However, the latest announcements represent much more than a simple name change and a new faceplate on the hardware.

If you haven't been following this (you have plenty to do keeping machines running from day-to-day, after all), then let me bring you up to speed. IBM is unifying its System i and System p systems onto common hardware platforms. The way the company puts it, there's a new power equation for the new enterprise data center: Power = i + p.

From IBM's announcement letter:

"The IBM System i and IBM System p organizations are unifying the value of their server offerings into a single, powerful lineup of Power Systems servers based on industry-leading IBM POWER6 processor technology with support for the IBM i (formerly known as i5/OS), IBM AIX, and Linux operating systems. This new, single portfolio of Power Systems servers offers industry-leading technology, continued IBM innovation, and the flexibility to deploy the operating system that your business requires.

"Specifically, being announced today are: IBM Power 520 Express, IBM Power 550 Express, IBM BladeCenter(R) JS12 Express blade server. All three of these systems can be ordered in the AIX Edition, i Edition or Linux Edition."

Then last week came another IBM announcement:

"IBM announced two high-end Power Systems models--the world's fastest UNIX server and a unique water-cooled supercomputer. The new systems offer sophisticated IBM virtualization technology and energy-saving capabilities to help dramatically reduce bottom-line operating costs, such as those for energy, floor space and systems management, while improving system performance, helping clients transition to a new enterprise data center. Beginning today, clients will be able to leverage the world's most powerful microprocessor, POWER6--with new world-record speeds of up to 5 GHz--in these new systems, leading to significant performance improvements across a wide array of applications.

"The new UNIX enterprise server, the Power 595, designed to extend IBM's leadership in the UNIX market, will be attractive to existing IBM clients as well as Sun Solaris and HP UNIX users. IBM's new POWER6 "hydro-cluster" supercomputer, the Power 575, is built to help users tackle some of the world's most challenging problems in fields such as energy, aerospace and weather modeling. The new super-dense system uses a unique, in-rack, water-cooling system and with 448 processor cores offers users nearly five times the performance and more than three times the energy efficiency of its predecessor, IBM's POWER5+ processor-based p5-575 supercomputer.

"The new IBM Power 570 is a unified version of the popular midrange POWER6 processor-based System p 570 and the System i 570. Existing customers can update to the new system at no-charge. The Power 570 runs any permutation and combination of i, AIX or Linux partitions offering the ultimate in flexibility and increased asset utilization and reuse. And with PowerVM, Power servers also run many Linux x86 applications."

So we'll be able to run i, AIX, Linux on Power, and Lx86 on everything from JS12 and JS22 blades, to the 520, 550, 570, 575 and 595 Power models.

What makes the most sense in your environment? A BladeCenter with some JS12 or JS22 blades running AIX, i or Linux? You could mix those blades in the same chassis with Intel or AMD blades to run whatever Windows or native x86 Linux applications you might require. Or maybe you need an IBM Power 595 running 5ghz POWER6 chips with 4 TB of RAM? You could carve that machine into LPARs running i, AIX and Linux as needed.

The same virtualization options are still available:

Using PowerVM for virtualization, we can "aggregate and manage resources via a consolidated, logical view."

What do these announcements mean for your organization? Again, the keyword is unification. You may have had separate System i and System p IT teams that managed their own hardware and operating system, with each always figuring that its platform was the best and the most important.

At a minimum, as new Power hardware arrives on the floor, I'd expect more communication between the teams. It makes sense to get some cross-training as we seek ways to make coexisting on the same hardware a reality.

If IT personnel don't communicate and make an effort to understand the other operating system, if organizations continue to maintain separate computing empires, the capability to run i and AIX on the same hardware will be useless. Although initially it may make sense in some environments to let each group run with its own hardware, that mentality will be harder to justify as management keeps hearing about virtualization and consolidation and the machines keep handling larger workloads.

Of course, since every shop has budget and hardware lifecycle concerns, not everyone will get the new Power hardware right away. But plenty of shops are struggling with older technology that is in need of a refresh. For those organizations that will soon go through the refresh process, be sure to look at the new machines and make the virtualization and consolidation decisions that are right for you.

Getting Hands-On with Live Partition Mobility

My first experience with live partition mobility came as an observer. A few months back I went to Austin and saw an LPAR move from one POWER6 570 to another.
 
Since then, I've become a live partition mobility user. I move running LPARs between my two JS22 POWER6 blades, which are connected to the same SAN and able to see the same LUNs. Once I got it set up, I haven't had any problems. It functions with the blades just as I saw it work on the larger systems.
 
Checklist
To get started, I made sure I had the enterprise edition of PowerVM. After loading the virtual I/O (VIO) server onto my blade and logging into the Integrated Virtualization Manager (IVM) GUI, I was able to enter my APV key.
 
I did some other checks during set-up. I made sure my network was set to use a shared ethernet adapter instead of a host ethernet adapter, and I made sure the reserve_lock on my SAN disk was set to no_reserve. (It was originally set to single_path.) If you fail to fix your reserve_lock, you'll need to change it. When searching on "change reserve lock," I found some documentation which included the following:
 
To change the setting with an LPAR running, from your vio server run:

# chdev -dev hdisk7 -attr reserve_policy=no_reserve
 
Some error messages may contain invalid information for the virtual I/O
server environment.
 
Method error (/usr/lib/methods/chgdisk):
    0514-062 Cannot perform the requested function because the specified
    device is busy.

The error makes sense because we've mapped the physical device to a virtual
device.

    $lsmap -vadapter vhost3
    SVSA            Physloc                                      Client Partition ID
    --------------- -------------------------------------------- ------------------
    vhost3          U7998.61X.100BB8A-V1-C17                     0x00000007
 
    VTD                   vtscsi3
    Status                Available
    LUN                   0x8100000000000000
    Backing device        hdisk7
    Physloc               U78A5.001.WIH0A68-P1-C6-T1-W5005076801303022-L6000000000000
 

Shutdown the LPAR that is using the device.

Remove the virtual adapter and its mapping.

$ rmdev -dev vhost3 -recursive
hdisk7 deleted
vhost3 deleted

Change the reserve_lock setting.

# chdev -l hdisk7 -a reserve_lock=no
hdisk7 changed

Let's make sure the setting changed.

$ oem_seup_env
(to leave the restricted shell)

# lsattr -El hdisk7
 
Check for
 
reserve_policy  no_reserve

With the changes verified, the next step is to re-create the virtual device using mkvdev.

Using my IVM GUI, I selected the LPAR that I was going to migrate. (In the drop-down menu, go to "mobility" and select "migrate.") I entered the IP address of the machine I was moving to, followed by the padmin password. Then I selected "validate." This verified that my LPAR was ready to move. Once it passed the tests, I clicked on the "migrate" option, and my LPAR moved to the other blade.   
 
Keep Running
The value of this technology became clear to me on a recent customer call. My customer was conducting maintenance. I had to bring the machine down, and the users suffered an outage. Had the customer been on POWER6, live partition mobility would have been a perfect solution here. I could have moved the running LPAR and then brought down the source machine without affecting the workload that was running.

I'm sure that as more customers deploy POWER6 technology, we'll see live partition mobility become more widely adopted. It's extremely useful technology.

Lx86 Works as Advertised

After hearing so much about Lx86 (formerly known as the System p Application Virtual Environment, or System p AVE), I finally decided to try it out.

Lx86 allows you to run unmodified Intel/x86 Linux binaries on IBM Power hardware. This is significant because the alternative--running Linux applications natively on Power--requires a recompile. This can be painful or impossible, and in fact it's this reason that many IBM customers choose Intel x86 to run their Linux applications.

For more about how Lx86 works, see the references at the end of this post. Now, I'll tell you about installing and getting started with Lx86.

First, I set up a trial subscription at Red Hat to get the .iso files I needed. There are 10 files in all--both 32-bit x86 images and 64-bit Power images. Once I downloaded the files, I moved the ppc .iso files to the /var/vio/VMLibrary directory on my virtual I/O (VIO) server, a process I cover here.

This allowed me to use my virtual optical drive to mount the images rather than burn and mount a bunch of physical media.
 
I booted from the first ppc .iso file, and after setting my install selections, I was able to install the machine onto an LPAR. I had to do some loadopt and unloadopt commands to get through all the CDs, but that seemed easier than personally taking a fistful of media to the site.
 
After the installation, I downloaded the code here. (IBM registration required.)
 
As of this writing I was using IBM PowerVM Lx86 V1.1 p-ave-1.1.0.0-1.tar (8652800).
 
I copied the tarball and my x86 .iso files to my newly built Linux on Power installation. I untarred the p-ave .tar file, and ran ./installer.pl. After accepting the license and registering with both IBM and Red Hat, I was ready to load the software. I chose to do a full install, and had it share my /home directories.
 
The installer prompts for the path to your x86 linux .iso files, as it expects to find them loaded on the machine. I gave the installer the correct path, and it examined the .iso files to verify it could find the rpm files it needed. I selected continue and it installed my x86 world. It basically copies all the necessary files to my /i386 directory, so that when you start the environment it can chroot into /i386 and function like this is the / directory.
 
After getting confirmation that System p AVE and x86 World were installed successfully, I was prompted to run /usr/local/bin/runx86 to start a shell. It returned to the menu and I selected Option 6 to quit. You can look at the install log by searching for p-ave_install*log.
 
I am now able to cd /i386 and runx86. Once I do this, the shell runs as it would on an x86 machine. If you run "arch" from your shell before you runx86, you'll see ppc. If you run arch after you're in your x86 shell, you'll see i686.
 
I can now run and install rpms and do anything else as if I were on a regular Linux machine. I ran vncserver and my desktop came up as I would have expected. I moved some rpm files into the environment and installed them with rpm -ivh as I normally would. There are caveats--this isn't a panacea where any and all applications will run at the same speeds as on native x86 hardware. But in many cases, the performance will be quite good.
 
I found that I wanted to ssh into the x86 World instead of the ppc world, so I opened my console, copied my /etc/ssh/ information from ppc world to x86 world, killed sshd that was running in ppc world, did a runx86 and started sshd from there. Once I did that I could ssh -Y into my x86 world and start exporting X applications to my desktop. I'll need to read more documentation and play with it more, but on first glance it works as advertised. I was able to simply install rpms and run them.

References

From:   
 
Transitive Corporation, a leading provider of cross-platform virtualization software that enables the execution of applications across diverse computing platforms, today announced that IBM will commence shipping PowerVM Lx86 with all copies of PowerVM Editions, available across its entire line of  System p servers. PowerVM Editions, a set of advanced virtualization offerings developed by IBM for Power Systems platforms, now includes the x86 feature (developed for IBM by Transitive) which simplifies migration of x86 Linux applications onto this popular platform for server consolidation and business application deployment. PowerVM Lx86 allows the creation of an x86 application virtual environment so users may easily install and run a wide range of x86 Linux applications on a Power Systems platform with a Linux for POWER operating system. PowerVM Lx86 allows thousands of x86 Linux binaries to run unmodified and without recompilation on System p servers, helping to bring additio!
nal benefits with IBM PowerVM virtualization to enterprise customers by enabling more applications to be consolidated.
 
From:
 
Up to now more than 2500 applications have been ported to Linux on POWER, but still there are thousands only ported to x86 based platforms. With the IBM PowerVM Lx86 environment, a customer can take the original installation media of a Linux on x86 application and install it as is within a Linux on POWER partition running on IBM System p. There are many workloads that will run well within this environment. There are a few workloads that are not recommended to be run in this environment
From a customer perspective, this environment allows a very transparent and easy way to start taking the benefits of such an advanced infrastructure platform. From an ISV perspective this environment provides an excellent opportunity for a jump-start onto a new marketplace, postponing the decision of the code porting from Linux on x86 to Linux on POWER to a more appropriate moment in time if he judges necessary. It also allows the ISV the opportunity for keeping his development and support costs on a lower level, since there is only a single source (x86-based) code.
 
From:
 
In addition, the performance of some x86 Linux applications running on PowerVM Lx86 may significantly vary from the performance obtained when these applications are run as a native port. There are various architectural differences between x86 and POWER processors which can impact performance of translated applications. For example, translating dynamically generated code like Java byte codes is an ongoing translation process, which can be expected to impact the performance of x86 Java applications using an x86 Java virtual machine. Floating point applications running under x86 have a different default precision level from Power Architecture processors, so translating between these levels can have additional performance penalties. And finally, translating and protecting multi-threaded applications can incur an additional performance overhead as the translator works to manage shared memory accesses. IBM suggests that clients carefully consider these performance characteristics when selecting the best method for enabling applications for their environment.