July 08, 2014

Webinars Cover the World of AIX

Hopefully you regularly listen to the AIX Virtual User Group webinars, either live or on replay. Recent sessions have been devoted to the POWER8 server announcements, Linux on Power and SRIOV.

If you're outside of the U.S., you should know that similar webinars are taking place worldwide. For instance, there's the IBM Power Systems technical webinar series that originates from the U.K. This group's next event, which is set for July 16, covers PowerVKM. Dr. Michael Perzl is the presenter, and as someone who's already working with PowerVKM, I look forward to what he has to say.

Previously, this group presented "More tricks of the Power Masters," which, as you might imagine, was an hour-long session consisting of tips and tricks for using IBM Power Systems hardware. Thirty-eight total replays of these sessions can be found here. Specifically, I recommend this video of several presentations by Gareth Coates. Gareth is an excellent speaker who's always on the lookout for tips he can use in future sessions, and he mentioned that he is on the lookout for IBM i content as well. (He'll be sure to give you credit for your help.)

As I've mentioned on numerous occasions, there's little I love more than learning, finding and sharing AIX tips and tricks. With that in mind, please indulge me while I cite some specific information that's available in the "Power Masters" videos:

* For starters, to force a refresh of the operating system level information on the HMC, run:


            lssyscfg –r lpar –m <managed system> --osrefresh


(In addition, Power Masters offers good info on performing HMC updates from the network, which I've also written about here and here.)

* To find out how many virtual processors are active on my system, use the kdb command (and use it carefully):

            echo vpm | kdb

* To protect AIX processes when AIX is out of memory, use:

            vmo –o nokilluid=X

* To test your RSCT connection, use:

          /usr/sbin/rsct/bin/rmcdomainstatus –s ctrmc

Some other Power Masters topics:

* Using Live Partition Mobility checklists. (I wanted to point this out so I have a reason to add that FLRT now has LPM checks available.)

* viosbr (which I've also covered here).

Some of the other information presented was first used in a session that took place in 2013, called Power "Ask the Experts." I covered that here.

Of course there's much, much more on not just AIX but also IBM i topics, so check out the Power Masters videos on YouTube. And if you don't already, be sure to tune into the AIX Virtual User Group and IBM Power Systems technical series webinars.

July 01, 2014

We're Not the Only Techies

As I've noted previously, I work with Boy Scouts. Recently I took a group of boys to an airport to work on their aviation merit badge.

We found a pilot who was willing and able to spend time on a Saturday with the troop. He invited the scouts to visit a maintenance and training facility and spend time on an airplane simulator.

Although he had interesting information to share, I quickly figured that, as a pilot, he hadn't spent a lot of time creating PowerPoint presentations. Prior to taking the scouts to the hangar so they could learn how to conduct a pre-flight inspection of an aircraft, he showed them a presentation covering the merit badge requirements. At one point, he clicked on what he hoped was a link to a video, but it turned out he had inadvertently made a screen capture of the video rather than an actual link to it. (Not that this issue wasn't easily addressed; he ended up going directly to YouTube and showing us things like this.)

But indeed, our pilot guide did admit that he hadn't used PowerPoint in years. On top of that, during the presentation, the overhead projector had an issue. For those of us who spend our time in meetings and conference rooms, fixing projector issues is second nature. Once again though, he wasn't immediately sure what to do.

All of us -- even the scouts themselves -- were pretty smug about our computer and projector knowledge at this point. Then we went into the next room and got into the simulator. Long story short:  I'm not cut out to land an airplane, or even to keep one riding smoothly through the air. So we all have our different skills. Frankly, as long as my pilots are experts at flying, I'll excuse their shortcomings when it comes to using software programs and projectors.

Of course the scouts, most of whom have considerable experience with computer games, made me feel even more inept on the simulator. A lot of those kids had a pretty light touch on the airplane controls and managed a reasonably good landing on the first try.

As an AIX pro, I'm generally surrounded by others with similar professional backgrounds. Quite possibly, it's the same for you. But we should all keep in mind that while most people need computers to do their jobs, they don't live and breathe technology the way that many of us do.

Ultimately, my day at the airport reminded me that, even if most people don't know computers like we do, we're far from the only smart folks out there doing challenging, technical work. And thank goodness for all these people and their unique specialties, because you really wouldn't want to see me at the controls of your plane.

June 24, 2014

More POWER8 Docs

I love reading about new computing technologies, particularly the latest IBM Power Systems releases. It doesn't hurt that, as a consultant, I have opportunities to work with the newest hardware, but even if that wasn't the case, I'd still want to know everything about what's coming out of IBM. I guess I'm like those folks who read automotive magazines, even though I don't plan on buying a new Tesla anytime soon.

With this in mind, I'd like to point you to three new IBM documents -- draft Redpapers -- that cover the recently unveiled POWER8 models.  All three publications are scheduled to be finalized by the end of this month.

As you might expect, given that the models have many of the same features, there's some overlap in the information presented. For instance, this is the table of contents for all three publications:

            Chapter 1. General description
            Chapter 2. Architecture and technical overview
            Chapter 3. Virtualization
            Chapter 4. Continuous availability and manageability

So if you read these Redpapers back to back, you might have a case of déjà vu. Nonetheless, I believe the information is well worth your time.

Let's start with redp5097, which covers the 4U models, the S814 and the S824. As a reminder, the S in the model number stands for scale out, the 8 stands for POWER8, the 1 or 2 stand for the number of sockets, and the 4 stands for 4U.

Redp5098 covers the S812L and the S822L. Again, as a reminder, S for scale out, 8 for POWER8, 1 or 2 for the number of sockets, and 2 for 2U. L designates that these are Linux-only servers. I wrote about my experiences with the S822L here.

Finally, there's redp5102, which covers the S822. For completeness, the S is scale out, the 8 is POWER8, the 2 is 2 socket and the 2 is 2U.

At the bottom of the splash page for each publication there's a link to a blog post that lists five things to know about the IBM POWER8 architecture. I suggest checking this out as well.

So what are your plans to run POWER8 in your shop?

June 19, 2014

Can Vendors Make Our Lives Easier?

Recently I was listening to a few admins compare and contrast two different shops that run AIX. All of them had worked in the first environment for several years. In this environment -- we'll refer to it as Shop No. 1 -- they were constantly fighting problems and fixing others' mistakes. Bad management and bad change control were cited as the primary issues. Pages would come at all hours of the day. Periodically missing a child's soccer game -- or a full night's sleep -- was the norm. Their jobs were stressful, to say the least.

Eventually, these admins found new positions working in customer environments where a vendor dictates much of their production computing environment. The vendor has very strict requirements that must be met before applications are allowed to run in production. Customers must have adequate hardware to handle the anticipated workload and capacity. Customer hardware must hit very specific IOPS numbers. The vendor requires access to customer systems that host the vendor software, and customers must agree to run vendor-specific monitoring tools on the systems. There are very strict requirements around change control, which means changes aren't made on the systems without approvals. It'd take a catastrophe -- a very rare and unusual event -- for an admin to ever get called out of bed to go to work. Understandably, this group of admins was happier, professionally and personally, working with this vendor's software. We could call this environment Shop No. 2.

Now for the discussion itself. The argument was made that if Shop No. 1 went with this vendor product or a similar solution that led to the same strict requirements being enforced, it would cease to be such a difficult place to work. But is it really possible that vendor requirements can so profoundly impact their customers' working environments?

From the vendor perspective, being hands-on makes sense. If I can get my customers to agree to run my software on systems that have the capacity and functionality to handle it, if I can get them to properly manage and monitor their systems, my software -- if it's any kind of quality product -- should work well. And I shouldn't have to deal with irate customers blaming me when they try to run my software on an under-capacity system or when their own in-house programming introduces bugs. It's in my interest to support customers -- and only those customers -- who agree to my requirements, because I can be confident there won't be issues. Why wouldn’t I want to do that?

On the flip side, if I'm an admin, why wouldn't I want to make sure my systems are capable of running the software I'm using? Why wouldn't I want proper change control processes to be instituted? Doesn't this seem like a win-win?

When considering vendors and the vendor solutions we deploy on our hardware, besides asking if a given software package will do the job, maybe we should ask how it will be supported. Because I can easily imagine a world where very specific instructions and very specific software support lead to very stable work environments.

How about you? Does your vendor have significant say-so over your environment, or is vendor input largely limited to implementation and support? How involved do you want your vendor to be?

June 10, 2014

Getting Started With PowerKVM

I recently installed and started playing with IBM PowerKVM software on the S822L. Luckily I had a good PowerKVM quick start guide to follow. There's also a draft version of a Redbook that will help you with your installation. It covers how to netboot and includes examples of the menus you'll encounter.

I started by connecting my laptop Ethernet port to the HMC1 port on the S822L. This port is using the default address of, while the HMC2 port is using the default of In my case I set my laptop to and logged into ASMI as I'm used to doing with PowerVM.

Once I was in I was prompted to change my ASMI password. Then I went to the system configuration/hypervisor configuration menu item in ASMI. There I was presented with a choice of using PowerVM or PowerKVM. I selected PowerKVM and entered an IPMI session password.

To get a console or power on and off the system, you need to get impitool. With a web search you can find methods using Cygwin to get it running on Windows, or you could look at ipmiutil. Since I was using Linux, I just made sure I had ipmitool installed, and I was set to go.

In ASMI I went to System information > Real-time progress indicator so I could see the LEDs from the front of the display without actually being in front of the machine. I verified I was at the 01 N V=N prompt, and then ran:

            ipmitool –I lanplus –H x.x.x.x –P password power on

Once I did this, I saw my LED codes change as the system powered on.

To get a console I ran:

            ipmitool –I lanplus –H x.x.x.x –P password sol activate

I soon discovered that I could kill my console session by running:

            Ipmitool –I lanplus –H x.x.x.x –P password sol deactivate

Finally, to power off I'd run:

            Ipmtool –I lanplus –H x.x.x.x –P password power off

I made sure the PowerKVM DVD (that I burned from an .iso image) was in the drive, the machine was powered up and my console was open. Eventually, petitboot came up. Due to a bad DVD I didn't initially get what I expected, but once I had a .iso image that was in good shape, I was able to select the PowerKVM LiveCD option. The screen displayed:


            System information

            System configuration

            Exit to shell


The install wizard (which reminded me quite a bit of an old school RedHat install) prompted me for a root password, time zones, which disks to use, etc. Once that was completed it installed

PowerKVM and the system rebooted. Then petitboot came back up. I selected my freshly installed system and I was able to boot to a root prompt.

During the install I specified a network address for my network card and made sure it was on the network. To get Kimchi to work I needed to get into my sol console and edit /etc/sysconfig/selinux, change the selinux permissions to permissive and then reboot the server.

I verified that kimchid was allowed in the firewall by running:


            firewall-cmd --list-services


I ran systemctl to see the state of Kimchi:


            kimchid.service             loaded active running   Kimchi server


Once in a while I'd see it wasn't running. In those instances I'd run:

            systemctl stop kimchid.service

            systemctl start kimchid.service


At this point I was able to connect to kimchi by going to

This gave me a graphical interface to simplify creating guests. I've gotten different versions of Redhat, SUSE, Ubuntu and Debian to all run successfully.

I copied my .iso files to /var/lib/libvirt/images and then set up templates in kimchi. It was pretty self explanatory. I'd click on the green + button, pick the local .iso image option and select create templates from selected .iso. At this point I could edit the number of CPUs, the amount of memory and disk, choose networking options, etc. Then I started my guest machine. I clicked on the live tile to get a console, and configured the machine as I would any new install.

By logging into my PowerKVM instance and running top, I can see all of the copies of my operating systems running as the qemu user. By running virsh commands I can get information about my machines as well as stop/start them, suspend them, etc. For example, this virsh command gave me information about the disks I was using for a machine called redhat7-1

            virsh qemu-monitor-command --hmp redhat7-1 info block

            drive-virtio-disk0: /var/lib/libvirt/images/9f82cbd6-d345-4591-aa68-748f2c7b2b4e-0.img (raw)

            drive-scsi0-0-0-2: /var/lib/libvirt/images/rhel-server-7.0-ppc64-dvd.iso (raw, read-only)

                Removable device: locked, tray closed


Another nice way to run the system is to simply enter virsh and run interactively. To see all the machines you can control, enter list. You can also get a console to a virtual machine -- in my case I entered:


console redhat7-1


This is definitely an interesting new way to access a Power Systems machine. As I continue to work with PowerKVM, I'll post more tips and tricks.