In IT, Right is Might

Having previously written about management in "A Tale of Two Managers," I really liked this Computerworld article. Some quotes:

"Geeks are smart and creative, but they are also egocentric, antisocial, managerially and business challenged, victim-prone, bullheaded and credit-whoring. To overcome these intractable behavioral deficits you must do X, Y and Z."

"... my personal experiences working within IT groups have always been quite good, working with IT pros for whom the negative stereotypes just don't seem to apply.”

"Few people notice this, but for IT groups respect is the currency of the realm. IT pros do not squander this currency. Those whom they do not believe are worthy of their respect might instead be treated to professional courtesy, a friendly demeanor or the acceptance of authority. Gaining respect is not a matter of being the boss and has nothing to do with being likeable or sociable; whether you talk, eat or smell right; or any measure that isn't directly related to the work."

There's much more, and I encourage that you read it all, but this idea struck me: I've worked with arrogant people, but I put up with them and their arrogant attitudes because they were really, really smart. I knew that when I asked a question, I'd get the right answer every time, end of story. Sure, I may have wished these people were wrong occasionally, but it never happened. They were always right, and being right is demands respect.

The article expresses it this way:

"While everyone would like to work for a nice person who is always right, IT pros will prefer a jerk who is always right over a nice person who is always wrong. Wrong creates unnecessary work, impossible situations and major failures. Wrong is evil, and it must be defeated. Capacity for technical reasoning trumps all other professional factors, period."

Fortunately, I have had the privilege of working with a few nice people who also knew their stuff. These people not only knew what they were talking about, they also patiently took the time to help others understand what they already knew. Yes, they were busy -- in part because they were constantly bombarded with questions from people who were unable or unwilling to find answers on their own. However, rather than berate their questioners, they took the time to help.

Are you a jerk who's always right? Are you nice but often wrong? Are you really nice and really good? Be honest.

Migrations Made Simple with NIM

I had some VIO servers running 1.5.2.5-FP-11.1 SP-01 that I wanted to upgrade to 2.1.2.10-FP-22. The machines were in a remote location and consisted of both blade and "regular" servers. While I could have recruited someone to burn the migration .iso image and physically install the media into each DVD drive, I wanted to keep things simple. I decided to use NIM for the migration.

It helped that I could follow this great document that outlines the necessary steps for conducting an NIM migration.

I started by downloading the migration .iso image from here and the VIOS fixpack from here.

Then I loaded the migration .iso image into a virtual optical library (something I discuss here).

After making the DVD image available to a client LPAR using my virtual optical device, I mounted the DVD in the client LPAR. Then I used NFS to export the mounted DVD from my client LPAR over to my NIM server, which in this case happened to be a stand-alone server. Obviously, if my NIM server had been a client LPAR of my VIO server, I could have used the virtual optical directly to my NIM server, but I had to get a bit more creative here. Ultimately though, I accomplished my goal, which was to be able to mount the DVD image on my NIM server.

I made sure to bring my NIM server to the latest AIX level (6100-04-01-0944 as of this writing) and then followed the instructions in the aforementioned Migrate VIO document.

In my case, I first ran:

"gencopy -d /mnt -t /export/vios all"

After everything was copied from the optical media, I ran these commands:

"nim -o define -t lpp_source -a location=/export/vios -a server=master
vios_migration_lpp"

“nim -o define -t spot -a location=/export/spot -a server=master –a
source=vios_migration_lpp vios_migration_spot”

The first command defined my lpp source; the second defined my spot.

At this point I logged onto my NIM server so I could set up my NIM client to install using rte. (By the way, here's more info on NIM.)

I rebooted the VIO server and went into SMS mode. After verifying that the network installation settings were correct, I booted the VIO server from the NIM server. Then I selected the option to have it migrate my VIO server. Everything came up and ran as expected. After the reboot, it reported that it was at level 2.1.0.0.

Then I loaded the fixpack by NFS mounting my fix directory and running:

updateios -accept -install -dev /nfsmnt/fixes/vios.fixpack.22

At this point, my VIO server reported that it was at level 2.1.2.10-FP-22. That's where I wanted it.

Physical media is fine; use it all you want. But I'll take virtual optical and NIM every time.

A 'Hot' New Option for Dealing with SSDs

For a while now customers have been asking me how to identify the filesystems and physical and logical volumes that are good candidates to move to solid state drives (SSDs). Given these drives' I/O characteristics and the higher prices they currently command, you should be selective about the data you move from traditional drives to SSDs.

Obviously you don't want to put seldom-used data on these pricey new drives--just leave that on your older, slower disks. Your SSDs should house critical, heavily used data. Fortunately, figuring out how to get the most from your SSDs just got easier.

The latest AIX releases--5.3 TL11 and 6.1 TL4--include new options that can help you determine the data that's best suited for SSDs. I updated some of my machines so I could test out these new options, and they seem to work as advertised.
 
The key new option is found with the filemon tool and its new flag, -O hot, which creates a report that shows you your most frequently accessed data.

To receive my filemon output, I ran:

filemon -O hot -A -x "sleep 20" -r fmon -o fmon.out

Incidentally, -O hot must run in conjunction with the -A flag to enable automated offline mode. It cannot run in realtime mode. I know this because I tried and received this error:

filemon -O hot  -o fmon.out
hot option not supported in realtime mode.

Once I had my output file, I ran trcrpt:

trcrpt -r fmon.out > fmon.rpt

Then I viewed a report showing me hot files, hot logical volumes, and hot physical volumes by running: more fmon.rpt

I intentionally ran the trace ran at a time where my disks were heavily utilized. Actually, it may make sense for you to run these traces several times, especially during peak workloads. This will give you a good idea about which files and physical and logical volumes may be good candidates for moving to SSDs.

This data will greatly help you make informed decisions about moving filesystems and physical and logical volumes to SSD.

While I do regularly work with and speak to customers, SSD is fairly new technology, so there's plenty more to be learned about how people are using these new drives. So please enlighten me and your fellow readers by sharing your experiences with SSDs in Comments.

On an unrelated note, I saw these interesting articles about tech support for the International Space Station that I thought I'd pass on.

Tools Gold

AIX pros have been using VNC and Screen for a long time. Years ago I wrote about these handy tools.

For a more recent and more in-depth look at Screen's capabilities, check out this article from IBM developerWorks.

I literally use Screen every day, yet I'm amazed whenever I find people who still don't know about it. I guess the methods and tools that I take for granted are always new to someone.

Another tool that I use daily is called Synergy.

I can remember showing a then co-worker how Synergy allowed me to have my Windows-connected mouse and keyboard also control a Linux machine via the network. (And, yes, the keyboard was a Model M. Of course it was.)

I had two systems, each with a monitor attached. Using Synergy, I can move my mouse seamlessly between the desktops, and even cut and paste between systems. This can work with multiple Windows machines, Linux machines or combinations of each.

I had a chuckle a few years later I visited that co-worker; he was still running Synergy and had been spreading the word. Now others in his office were using it, too.

I always like hearing about how people use their systems. How are they being backed them up? How many monitors are there, and are they extended desktops or different machines, with each running synergy? Are vnc and screen being used? What different things are being done on the command line?

For chatting, I like pidgin.

From Wikipedia:

"Pidgin (formerly named Gaim) is a multi-platform instant messaging client, based on a library named libpurple. Libpurple has support for many commonly used instant messaging protocols, allowing the user to log into various different services from one application."

For a good ssh/scp/sftp client on Windows, there's PuTTY, which, as explained here, is "a free implementation of Telnet and SSH for Win32 and UNIX platforms, along with an xterm terminal emulator. It is written and maintained primarily by Simon Tatham."

Whenever I'm forced to use a Windows workstation, I rely on several options that will give me a UNIX-like experience, including cygwin, VMware (also here) and Damn Small Linux (DSL).

As explained here, DSL is "a very versatile 50MB mini desktop-oriented Linux distribution. Damn Small is small enough and smart enough to... boot from a business card CD as a live Linux distribution (LiveCD), boot from a USB pen drive, boot from within a host operating system (that's right, it can run *inside* Windows)."

Which tools make your life easier? Please share your experiences in Comments.

More From the Grab Bag

Every now and again I like to present a grab bag of links that I find to be interesting, even if they don't warrant their own post.

For instance, some weeks back I wrote that more people are talking about AIX on Power and its industry-best downtime numbers.

Here's more on that topic.

"Among the customers surveyed by ITIC, IBM's Power Systems running AIX experienced (this includes older System p and pSeries iron) the least amount of downtime per year, when averaged across all customers using these platforms. AIX shops reported an average of 0.42 Tier 1 incidents per year and 0.34 Tier 2 incidents, and not one customer reported a Tier 3 outage on their AIX boxes. The Power Systems machines (and this includes older System i and iSeries iron) had an average of 0.56 Tier 1 outages per year, 0.44 Tier 2 outages per year, and 0.12 Tier 3 outages. So in 2009 at least, the i platform fared a little worse than the AIX platform running on Power iron."

I also liked these recent articles:

* From ComputerWorld:

"The National Oceanic and Atmospheric Administration (NOAA) system, which went online in August, is compromised of 'Stratus' and its backup, 'Cirrus,' two separate supercomputers with about 5,000 IBM POWER6 chips running AIX operating systems."

* IBM developerWorks has this piece on a script that can gather information about your systems. Then you run the diff command to find the differences between them.


* Here, the reuse of SAN disks is examined:

"The support of these scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported has never been officially documented. There have been some documents and IBM Redbooks that have claimed support for specific scenarios, but they do not list the specific steps or restrictions. The scenarios detailed here guide systems administrators through the steps taken to achieve the specific environment desired. They also attempt to explain why the setup must be followed to achieve the desired results. If the steps are not followed, in some cases the system may not boot."

* Here's an IBM Systems Magazine Web Exclusive on using the mkramdsk command:

"One of the most common concerns for system administrators is maximizing disk performance. The AIX command mkramdisk is ideal for producing very high speed I/O by letting the memory do all the work. Database administrators are well aware of the benefits of keeping frequently accessed data in memory in order to reduce the need to retrieve that data from disk. The AIX mkramdisk command allows system administrators to create memory-resident file systems. The performance benefits of using RAM disk can be astonishing. The unload of a large TSM database was reduced from 40 hours on SAN disks down to 10 minutes using RAM disk."

* Finally, here's something to look forward to with the POWER7 processors:

"IBM's Hot Chips presentation on its forthcoming 45nm POWER7 server processor had a wealth of information on the chip… POWER7 will come in 4-, 6-, and 8-core varieties, with the default presumably being the 8-core and the lower-core variants being offered to improve yields."

If you've found interesting articles online, please share your links in Comments.