The Hard Lessons of IT

When I was 16, I got my driver's license. One summer I had the opportunity to live in another state. However, my primary vehicle, a 1972 VW Bug, had to stay put. Still, I figured I needed wheels for that summer, so, after a year of busing tables at a Sizzler Steak House (and riding my bike to work), I saved enough to shell out $200 on a used station wagon.

It's easy to understand why teenaged me made that choice. Speed and power. I seem to remember being told that the wagon had a "Buick 454." I knew just enough about cars to understand that it had eight cylinders and that no one would be able to catch me in it. My friends' sports cars looked better, but they couldn't touch my beat up old station wagon. I never lost a race. (Of course it helped that none of my friends had a REAL sports car, but that old wagon was pretty fast.)

Still, as you might imagine, that purchase ended up being a very costly decision. First, I had to take it through state vehicle inspection, where issues with the muffler and exhaust system were found. I paid for the needed repairs (which, if memory serves me, cost at least as much as I'd spent on the wagon itself) and passed inspection. I thought I was ready to bask in my personal Summer of George, but my vehicle issues were only beginning.

The wagon burned oil--as much as a quart or two for every tank of gas I'd consume. And, if you weren't already thinking it, the gas mileage on this thing was miniscule. So every few days that summer, I was buying oil and gas. I figured I had enough cash on hand to coast through those two months, but before long, I was almost tapped.

Eventually, I only had enough cash on hand for gas or oil. One, not both. As a result, my bad decision to get the wagon in the first place would be compounded by an even worse decision. I chose gas over oil. In no time at all that wagon ended up stalled on the side of the road, having thrown a rod. Then it was off to the junkyard.

Oh, and I made one other bad decision. Before my wagon was junk, I could have sold it. Someone was pestering me to buy it, and my plan was to unload it on him at summer's end. Needless to say, that plan blew up with the motor.

I did at least learn from my mistakes. (For starters, it's oil over gas. Always.) I guess that's the important thing. If anything, it's even more important now. Mistakes surely happen in IT, some even more costly than blowing an engine. Sometimes a backup doesn't get made, and a system can't be restored. Sometimes a backup is made but not tested, and thus doesn't work when it's needed. Sometimes shutdown –Fr gets run on the production box rather than the test box you thought you were logged into. Sometimes a VIO server is misconfigured and the production network goes down. Sometimes an rm command is run in the wrong directory.

These experiences lead to outages, and probably a few lost jobs. But the culprits undoubtedly learned from their mistakes.

What mistakes in IT have you made and, hopefully, learned from? I love hearing others' IT horror stories (in part, admittedly, because they didn't happen to me). So if you'll indulge me, please share your learning experiences in Comments. I'm sure other readers will appreciate your stories as much as I would. And maybe by sharing you'll keep someone else from making the same error.

What to Say About What You Do

This recent Anthony English post got me thinking. When someone asked him what he did, he wasn't sure how to respond. How do you answer that question? Can you explain what you do in a nice 30-second elevator pitch?

Luckily for me, Watson recently made a repeat appearance on "Jeopardy!" Watson has become the basis for my "pitch": I simply tell people I work on the same servers that they saw on TV with Alex Trebek.

Of course, I'll never be able to count on "my" servers appearing on television with any regularity, so I still need other ways to explain to others how I make my living. Or do I? As I've noted previously, telling people you work on computers can have some unwanted consequences. In their minds my admission is their opening to ask me to come over and resurrect their old, possibly virus-ridden machines. ("Can’t you just add a hard drive or memory or something?") I generally counter those requests by explaining that I work on large enterprise servers running enterprise operating systems--in other words, small machines aren't my specialty. Of course to a lot of folks, a computer is a computer.

(As an aside, why is it considered bad form to ask your doctor and lawyer acquaintances for free legal or medical advice, but few think twice about asking computer nerds they know for free help? Maybe they figure we have nothing better to do. Maybe we need to start quoting our hourly rates.)

So what do you say when you're asked what you do? Do you talk about a typical day at work (one where machines haven't blown up)? I've heard a system administrator liken his job to a plumber's: nobody notices or needs one until something stinks. (That's figuratively, one can only hope, in the admin's case.)

If you're reading this blog, I figure you're involved with provisioning machines. You may not have built them, but you do care for them day to day. Interacting with your machines as much as you do, over time you may even get close to them. However, eventually you must put them out of their misery and upgrade to newer gear. (Another aside: It's actually funny how quickly this cycle can run. You just installed the fastest, shiniest new hardware, but in a few short years you're yearning for that new, state-of-the-art box.)

Ultimately, the way I explain what I do depends entirely on my audience and their frame of reference. To those in the industry, it's simple: I start by saying I sell and install IBM's Power hardware line and specialize in AIX. But to non-industry folks, I can't have their eyes glazing over from my tales of patching, upgrading, installing, cabling, provisioning and deploying machines. Not to mention backups, restores, clones, LUNs, mirrors, migrations, copies, archives, scripts, cron jobs and the like. Maybe it's enough for them to know that my specialty is enterprise IBM hardware. Or maybe I just say I work with computers.

What do you do?

Media Makes the Difference

A customer recently called because they couldn't login to their machine. A new server was being built, and someone had rebooted the virtual machine. Once the system came back up, no one could ssh or telnet to it, though they were able to ping it across the network.

I was in a location that allowed me to set up webex. This way, we could both see what was going on instead of me simply hearing about it over the phone.

We started by running putty and making an ssh connection to the HMC. From there, we ran vtmenu, chose a frame and selected the LPAR on that frame. We were able to open the console window, and we had a login prompt. However, we couldn't login as root. We tried a few different combinations of user IDs and passwords, but no luck. The machine appeared responsive, though. Had someone changed the passwords?

The decision was made to reboot the machine and login in maintenance mode. This way we could change the root password and get logged in to verify the network communications.

Because this environment wasn't virtualized, it wasn't as easy as simply booting from a virtual optical disk. We also discovered that the NIM server lived on this non booting LPAR, so booting from NIM to get into maintenance mode wasn't going to work.

Luckily the disk controller that the CD was attached to was available, so we made the controller and the CD available to this LPAR and had someone load the physical AIX DVD into the drive. We booted the LPAR into SMS mode and then selected the correct CD device to boot the machine. Instead of choosing to install AIX, we started maintenance mode for system recovery. Then we chose to access a root volume group and start a shell.

Now we were logged in as root, and we were able to poke around. The filesystems looked OK after running a df, but when we tried to run the passwd command, we got an error. Everything pointed to a corrupt /etc/passwd file, but when we attempted to look at that file, we found that it didn't exist. Someone had accidentally wiped it out. However, because /etc/security/passwd still existed their passwords were still there, and we just needed to get a copy of /etc/passwd back into the system. Once we did so and rebooted the machine, it came right up and we could login.

We did see a few rm –rf commands in .sh_history, but we didn't find the actual smoking gun to prove that the file was deleted. We did learn though that someone was copying /etc/passwd files around the environment, so it was certainly possible that this person erred when manipulating the files.

So how is your environment set up? Are you taking mksysbs? Are you backing up individual files so that you can recover them if needed? Do you have a NIM server available to boot and restore from? Do you have install media handy that you can boot from? Install media was the key in this case. Although my customer's problem was fairly trivial and relatively easy to fix, having the install media on hand allowed us to resolve the issue quickly.

The Delicate Art of VIOS Configuration

What's the quickest way to get to know your network team? Just bring down the entire network.

I actually know of people who have caused network outages by misconfiguring dual VIOS. However, this isn't another of my scary stories--I just want to tell you how to avoid stirring up your own broadcast network storm.

Start with this sample example:

mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 2 -attr ha_mode=auto ctl_chan=ent1

When you run this command, make sure that each VIOS is set up to use the same control channel VLAN (ent1 in this case). If not, the two servers will be unable to communicate with one another. And if that happens, each will respond as if the other VIOS is down, and each will attempt to function as the primary server.

From IBM Support:

"A Shared Ethernet Adapter (SEA) can be used to connect a physical network to a virtual Ethernet network. It provides the ability for several client partitions to share one physical adapter. SEA can only be configured on the Virtual I/O Server (VIOS) and requires the POWER Hypervisor and Advanced POWER Virtualization feature. The SEA, hosted on the VIOS, acts as a Layer-2 bridge between the internal and external network.

"One SEA on one VIOS acts as the primary (active) adapter and the second SEA on the second VIOS acts as a backup (standby) adapter. Each SEA must have at least one virtual Ethernet adapter with the
'Access external network' flag (previously known as trunk flag) checked. This enables the SEA to provide bridging functionality between the two VIO servers.

"This adapter on both the SEAs has the same PVID, but will have a different priority value. A SEA in ha_mode (Failover mode) might have more than one trunk adapters, in which case all should have the same priority value. The priority value defines which of the two SEAs will be the primary and which will be the backup. The lower the priority value, the higher the priority -- e.g. an adapter with priority 1 will have the highest priority. An additional virtual Ethernet adapter, which belongs to a unique VLAN on the system, is used to create the control channel between the SEAs, and must be specified in each SEA when configured in ha_mode. The purpose of this control channel is to communicate between the two SEA adapters to determine when a failover should take place."

In other words: When setting up VIOS, you must set up a control channel so that the two servers can communicate with one another. You also need to establish one VIOS as the primary server and the other as the backup.

This document states the consequences of misconfiguring your SEAs:

"In this section, you will create the control channel virtual Ethernet adapters on VIOS1 and VIOS2, which will communicate on VLAN ID 12. It is very important to create this adapter on both VIOS partitions before creating SEA adapters to support failover for the same VLAN. Failing to have proper control channel configuration can result in causing a broadcast storm when both SEA adapters are activated on the same VLAN (VLAN ID 2 in this case).

"First you will create the control channel adapters on each VIOS partition. These control channel adapters are used to determine the health of the SEAs and are required to avoid a broadcast storm (which can result when two trunking virtual adapters are available on the same VLAN)."

In another part of this document, we read:

"Failing to have proper control channel configuration can result in causing a broadcast storm when both SEA adapters are activated on the same VLAN (VLAN ID 2 in this case)."

And again:

"When you run the mkvdev -sea command, it is very important that you specify the ha_mode and ctl_chan attributes. If you fail to do this, creation of the primary adapter on VIOS2 could result in a network broadcast storm."

And again:

"STOP!!! Before you continue to the next step, ask a lab instructor to determine that you have the correct adapter configuration. Failure to properly configure an SEA failover scenario can result in a broadcast storm than can affect the entire lab network."

A network guy I know recommends enabling BPDU on our Cisco switches to try to address this issue. This website seems to agree with that assessment:

"As a precaution, you can enable Bridge Protocol Data Unit (BPDU) Guard on the switch ports connected to the physical adapters of the SEA. BPDU Guard detects looped Spanning Tree Protocol BPDU packets and shuts down the port. This helps prevent broadcast storms on the network."

Maybe some networking gurus out there can let us know whether using BPDU is advisable on our VIOS-connected ports.

Even those of us who routinely work with VIOS shouldn't get cocky, because one wrong move can take out a network. So be careful. The stakes are high.

Getting Started With AIX 7.1

Well, I was wrong. After arguing in two posts (here and here) that getting physical media from IBM is preferable to downloading AIX images, I am now among the converted. Sort of.

What happened? AIX 7.1 happened. When it was released on Sept. 10, I just had to get my hands on it. That meant I had to download it. Don't worry; I also ordered my physical media. But I did download a copy so I could work with it right away.

Glad as I was to get started with the newest AIX version, the experience reminded me why I'm generally reluctant to download these files. I couldn't unzip one file,
AIX_7.1_Base_Operating_System_TL_7100-00-00_DVD_1_of_2_092010.iso.ZIP. Instead, I received this error message:

End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a     multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of
AIX_7.1_Base_Operating_System_TL_7100-00-00_DVD_1_of_2_092010.iso.ZIP
or
AIX_7.1_Base_Operating_System_TL_7100-00-00_DVD_1_of_2_092010.iso.ZIP.zip,
and cannot find
AIX_7.1_Base_Operating_System_TL_7100-00-00_DVD_1_of_2_092010.iso.ZIP.ZIP, period.

Since I was able to unzip the AIX_7.1_Base_Operating_System_TL_7100-00-00_DVD_2_of_2_092010.iso.ZIP (disk 2) file with no trouble, I thought maybe the download didn't complete the first time. So I tried downloading the whole thing again, with the same results. Interestingly though, once I moved the file to an LPAR running RedHat Linux on the same frame, it unzipped just fine.

After completing the download and the decompressing the files, I moved the .iso images over to my virtual media repository, booted an LPAR from it and loaded AIX 7.1. I was able to select the edition of AIX I wanted to install, and I was able to navigate through menus and pick different software install options, including my preferred browser and the server packages I wanted to install. It looked like a normal AIX installation.

     1  System Settings:
         Method of Installation............. New and Complete Overwrite     
         Disk Where You Want to Install.....hdisk0

    2  Primary Language Environment Settings (AFTER Install):
         Cultural Convention................English (United States)
         Language ..........................English (United States)
         Keyboard ..........................English (United States)
         Keyboard Type......................Default
    3  Security Model.......................Default                  
    4  More Options  (Software install options)
    5  Select Edition.......................express

Install Options

 1.  Graphics Software................................................ Yes
 2.  System Management Client Software................................ Yes
 3.  Enable System Backups to install any system...................... Yes
     (Installs all devices)
4.    Import User Volume Groups........................................ Yes

Install More Software

 1. Firefox (Firefox CD).............................................. No
 2. Kerberos_5 (Expansion Pack)....................................... No
 3. Server (Volume 2)................................................. No

Nevertheless, I have a couple of nits to pick: First, why is the Manage Editions menu option so high up on the SMIT main menu?

  Software Installation and Maintenance
  Software License Management
  Manage Editions
  Devices
  System Storage Management (Physical & Logical Storage)
  Security & Users
  Communications Applications and Services
  Workload Partition Administration
  Print Spooling
  Advanced Accounting
  Problem Determination
  Performance & Resource Scheduling
  System Environments
  Processes & Subsystems
  Applications
  Installation Assistant
  Electronic Service Agent
  Using SMIT (information only)

Even computer geeks build muscle memory, and we've become used to the devices or system storage manager options being only a couple of down arrow keystrokes away. Now we must unlearn years and years of keyboarding. It's the new options -- especially those that probably won't change often -- that should be buried further down the list.

So AIX 7 was loaded, and rebooted. Now we come to my second issue: Having to watch this message on my console as I waited for it to allow me to login to the system for the first time:

    This is the first time starting Director Agent. Please wait several minutes for the initial setup...

    Stopping The LWI Nonstop Profile...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...
    Waiting for The LWI Nonstop Profile to exit...

It'd be nice if I could start the System Director Agent when I wanted to, or at least have the option of running it in the background while I install systems.

And of course, ssh wasn't installed by default, but as always it was easy enough to install it after the fact.

Another thing keep in mind is that if you plan on working with versioned WPARs (where AIX 5.2 can run in a WPAR), those vwpar filesets aren't loaded by default. Be sure you've ordered them separately.

Along with the unveiling of AIX 7.1, new VIOS code was released. Read about Anthony English's experiences with VIOS. (And once you've done so, check out Anthony's comments about migrating the NIM server to AIX 7.1.)

Finally, please share your experiences with AIX 7 in Comments. While I realize that many of you aren't yet considering migration, I am curious to know if AIX 7 is in your near-term plans.

The Downside of Uptime

A customer recently performed some scheduled maintenance on a critical server that had an uptime of nearly two years. The customer had created some great scripts that would bring down the application and then connect via ssh to the database server to bring down the database. The application start scripts worked the same way -- they'd remotely connect to a database server and bring up the database during the application startup process.

After successfully completing the server maintenance, it was time to bring the application back up. The customer ran the application startup script, but the application didn't appear to be working properly. After some phone calls to application and database support personnel, it was determined that someone had commented out a line in the startup script. The line that was commented out was the command that would ssh to the database server to start the database, and the application relied on the database in order to work properly.

I've said it before: When making changes to a machine, the changes must be tested. Again, in this case, the timestamp on the changed file was nearly two years old. So the change was made, it was never tested, and it was forgotten about. It could have been a simple case of testing something else in the script that affected the startup process and not wanting the script to contact the database server, but once that testing was done, uncommenting the line was forgotten. Since the timestamp was so old, it wasn't a smoking gun. It didn't stand out when troubleshooting was done on the issue, so it took a while for someone to actually check the script to see that it did what it should be doing. People assumed that such an old startup script had not been changed, so it should still be working as it would have been used at some point over the last few years.

Although none of us like downtime, especially with resilient servers that "just run," maintenance windows and application restarts are well worth doing. If we don't regularly exercise our server shutdowns and startups, we may not uncover a script problem or some other issue until long after the change is made. But by scheduling reboots each month or each quarter, these changes will be more quickly detected and dealt with.

The same holds true with IBM PowerHA clusters. I always like to know that a failover is being regularly tested. The wrong time to find out that something doesn't work is when the application actually needs to failover.

Having machines that can stay up for years is a tremendous thing. But there's nothing like the peace of mind that comes from knowing your machines stop and start the way they're supposed to.

Think of Our Users

When I previously wrote about using my BlackBerry, I mentioned how it helps me in my travels.

I use my phone as my GPS, having it give me turn-by-turn audio directions to the different places I go. I travel quite a bit, and find myself in unfamiliar cities. I usually program the different addresses that I need before leaving, then bring them up on my phone once I reach my destination.

It's been great for me. Of course I'm taking a chance that the GPS receiver and phone will always work, and that I'll have network coverage wherever it is I am. However, I haven't had any of those issues. But the one problem I have experienced is the one I didn't expect--a failure with the application vendor as the source.

Whenever I drive a long distance, I usually wait to activate GPS. Generally I don't need directions for the entire trip, just the very end of it. Why use GPS for, say, all 120 miles when I really only need it for the final five?

So recently I'm on the road and getting close to my destination. When I activated the application, I was informed of a critical update. I had no idea what the critical fixes were, and I only received two options: download the new application or quit. I would have preferred to download the update later, but no such luck.

I decided to pull off of the highway and download the application. It was a relatively smooth process. I was worried that I might have to re-enter an authorization code, but the application came right up when I restarted it. However, all of the addresses that I'd entered were lost.

It wasn't a disaster. I had plenty of time, and I managed to find the one address I needed at that moment and plug it back in. But had I been in a rush and lacking ready access to the address of my destination, I could have been in some trouble.

This experience got me thinking about the testing that went into this code deployment. Did the application vendor really not consider the urgent needs of GPS users? How can you force someone who's in transit to do an on-the-spot upgrade? At a minimum, users should have the option to perform such upgrades at their convenience. I know I like to upgrade when I have the time to work on any issues that might arise, not when I'm racing down a highway.

And thinking about that got me thinking about planned downtime and our own users. Do we administrators give them adequate notice and advanced warning about the updates we apply and the system changes we make? Do we test and verify in the lab, or do we push these changes without considering the consequences?

A friend keeps analog backups before he goes anywhere. He'll print out the directions and the addresses he needs--just in case his technology lets him down. Do we admins take similar precautions? Before making any changes, do we make certain we have adequate hard copy documentation and sufficient backups, just in case our upgrades go bad?

The experience with my GPS was only a minor annoyance, but I wonder how many others have landed in a world of hurt by performing forced upgrades on the fly and losing all their information at the same time.

Let's think of our users--our customers--whenever we make decisions that impact them.

The Case for Standardization

This fall I attended the RedHat conference. The conference hosted a break fix challenge, and the outcome was interesting.

Consider two particular teams of administators that participated in the challenge. One team came from a large company that adheres to very strict server build and maintenance standards. Each machine its administrators touch is identical; there are no deviations. Every node admins login to is the same as every other node. You can see how this makes administration easier. When all of your tools, logs and cron jobs are in the same place, running at the same time, troubleshooting can be simplified. When you work in the same environment every day, you become very good at fixing that set of machines.

Another team came from a large company that manages machines for customers. When something breaks, these admins literally have no idea what they might find. Where are the tools? Where did the customer load the scripts? Which jobs are running when? When these folks get a call, they must spend some time getting familiar with the environment.

You can imagine which team fared better in the break fix challenge. Although I'm sure there are shades of gray when describing these administrators, the guys that constantly go into unknown environments to solve problems prevailed.

But what about administering your environment? Would you rather have the admins from the first team or the second team? You might think that there's no need for standard server builds. You want to keep your guys sharp. You want them on their toes. You want them to be able to figure out the environment before they go fix things.

While this makes sense when you're managing machines for customers, when they're your own machines, I certainly prefer a standardized approach. I know when I've worked across large teams where we shared pager duty, it made life easier knowing I could log into a machine and find things where I expected to find them. Solving problems was quicker. I could get back to sleep sooner.

I've said it before (here and here), and I continue to believe that companies benefit from having good server build documentation and procedures.

For instance, I was recently at a customer site. The company had had some staff turnover, and the machines were set up in different ways. This caused nothing but headaches for the new guys who were trying to figure out the environment and prioritize the projects.

Standardization does make life easier, both for the teams that we work on now, and those who will follow in our footsteps. So let's have good documentation, and good server build documentation. It's fun to sharpen our skills to win a contest, but let's make sure we're sharpest where it counts. Let's build the perfect automated system to turn out identically built servers.

A 'Hot' New Option for Dealing with SSDs

For a while now customers have been asking me how to identify the filesystems and physical and logical volumes that are good candidates to move to solid state drives (SSDs). Given these drives' I/O characteristics and the higher prices they currently command, you should be selective about the data you move from traditional drives to SSDs.

Obviously you don't want to put seldom-used data on these pricey new drives--just leave that on your older, slower disks. Your SSDs should house critical, heavily used data. Fortunately, figuring out how to get the most from your SSDs just got easier.

The latest AIX releases--5.3 TL11 and 6.1 TL4--include new options that can help you determine the data that's best suited for SSDs. I updated some of my machines so I could test out these new options, and they seem to work as advertised.
 
The key new option is found with the filemon tool and its new flag, -O hot, which creates a report that shows you your most frequently accessed data.

To receive my filemon output, I ran:

filemon -O hot -A -x "sleep 20" -r fmon -o fmon.out

Incidentally, -O hot must run in conjunction with the -A flag to enable automated offline mode. It cannot run in realtime mode. I know this because I tried and received this error:

filemon -O hot  -o fmon.out
hot option not supported in realtime mode.

Once I had my output file, I ran trcrpt:

trcrpt -r fmon.out > fmon.rpt

Then I viewed a report showing me hot files, hot logical volumes, and hot physical volumes by running: more fmon.rpt

I intentionally ran the trace ran at a time where my disks were heavily utilized. Actually, it may make sense for you to run these traces several times, especially during peak workloads. This will give you a good idea about which files and physical and logical volumes may be good candidates for moving to SSDs.

This data will greatly help you make informed decisions about moving filesystems and physical and logical volumes to SSD.

While I do regularly work with and speak to customers, SSD is fairly new technology, so there's plenty more to be learned about how people are using these new drives. So please enlighten me and your fellow readers by sharing your experiences with SSDs in Comments.

On an unrelated note, I saw these interesting articles about tech support for the International Space Station that I thought I'd pass on.

Changing the padmin .profile

Recently I worked with a customer who was frustrated with the VIO server. When he logged in as padmin, he was faced with a $ prompt. When he ran oem_setup_env and became root, he had a # prompt. Whenever he switched users, he had to remember to run set –o vi to get access to his command history. He would log into two VIO servers and couldn't easily keep track of the one he was working in. For all of the other machines in his environment, he had customized his .profile to his liking. He customized his prompt and the name on his title bar, and would automatically run set –o vi, etc. However, those oddball VIO servers weren't playing nice.

I know that we're supposed to treat the VIO server like an appliance. We're supposed to set it up and let it run. But in this instance, we were still in the middle of setup. We were mapping LUNs to different vhost adapters for different clients, and we were going into oem_setup_env to install additional drivers.

Often I see customers set up their .profile with things like set –o vi, or customize their prompts with things like the user ID they're logged in with, their current working directory, their hostname, etc.

While you can certainly make changes to the /home/padmin/.profile file, they will usually go away whenever you upgrade your VIO server. That's fine; since we're supposed to treat it like an appliance, I understand that we shouldn't change things. In reality though, I continually find customers who alter the padmin .profile.

Even when customers change the .profile, those changes don't carry over when running the oem_setup_env command. For the sake of usability, something had to give in this case. Whenever we ran the oem_setup_env command, we had to run manually run /.profile. This customer was getting fed up.

After asking around for a more elegant solution, someone reminded me to just use a .kshrc file.

I was told to create a /home/padmin/.kshrc file. I put set –o vi and my other desired prompt settings into that file. Then at the end of the /home/padmin/.profile, I added:

export ENV=/home/padmin/.kshrc

This allowed the customer's desired environment to be set up automatically when he logged in as padmin. Then when he switched over to root by running the oem_setup_env command, that was set up as he wanted it.

Yes, I realize that .profile and .kshrc will have to be revisited when this customer upgrades his VIO server. But in the meantime, this technique allowed us to customize the environment to his liking. It helped me keep a customer happy. Maybe it will make you happy as well.