IBM Sticks with the HMC

So the SDMC evolution was upon us. I took my test drive. But then, just like that, it's over. The HMC has apparently survived.

Nigel Griffiths (mr_nmon) posted these tweets:

• "SDMC withdrawn as IBM listens to customers. SDMC only functions like dual VIOS+AME for Blades go in the next HMC version. Long live the HMC."
• "I had better add: SDMC owners needing Blade functions it is supported for 2 more yrs & you can convert to HMC (same HW) at your convenience."

 What exactly is going on?

Start with these announcement letters (here and here). IBM is no longer selling SDMC units. The SDMC hardware indicator is going away. The SDMC and SDMC virtual appliances are also going away, although they'll be supported through April 2015.

IBM has announced this statement of direction:

"IBM intends to enhance its systems management capabilities for Power Systems hardware as follows:

• Continued integration between the base platform management capabilities in HMC and advanced capabilities in IBM Systems Director
• Enhancement of the HMC to add support for Power Systems blades and mixed rack and blade server environments
• New HMC virtual appliance offering
• New process for transitioning from SDMC to HMC
• Improved usability"

I, for one, look forward to running the HMC appliance in smaller environments.

This FAQ includes the following, but read the entire document as I didn't copy everything:

"Why is this change being made?

"Clients have asked IBM to continue enhancing the HMC, which is a trusted, secure and dedicated management appliance for Power Systems. At the same time, due to the rapid increase in adoption of advanced virtualization and cloud solutions by Power Systems clients, IBM plans to continue enhancing the virtualization management capabilities of the Systems Director management server software.

"What are the options for clients using an SDMC?

"Clients currently using an SDMC for management can either convert it to an HMC (at no charge) immediately, or continue using the SDMC and convert it at a more convenient time in the future. The SDMC will continue to be supported by IBM until April 2015.

"What are the options for clients using Power blades?

"Clients currently using an SDMC for managing Power blades can continue using it or switch to using IVM instead. When the new release of the HMC (featuring blade support) is available, they can then convert the SDMC to an HMC at no charge.

"Are hardware changes required to convert an SDMC to an HMC?

"The SDMC is based on the same hardware as the HMC, and no hardware changes are required to make the conversion. After converting an SDMC to an HMC, clients can benefit from the additional memory and disk capacity.

"Will endpoint management licenses be required for the HMC?

"No.

"What happens to support agreements after clients convert an SDMC into an HMC?

"SDMC clients with current SWMA contracts may contact IBM Support for assistance before, during and after the HMC conversion process. When their SWMA is due for renewal, clients who have converted an SDMC to HMC should renew using the HMC Machine Control Program Remote Support
Agreement (MCRSA).

"How can a client using SDMC to manage Power blades transition to using IVM?

"As an alternative to the SDMC, clients can manage Power blades with the IVM. However, the IVM has limitations (compared to the SDMC and HMC) and cannot support dual VIOS configurations.

This IBM Redpaper covers SDMC-to-HMC migrations.

Basically you prepare for the migration by gathering IP addresses, verifying passwords, backing up profiles and then removing managed systems and frames from the SDMC. You download the HMC code and service packs (or order it on media) and then install the HMC code. I understand IBM will soon come out with video demonstrations of this process.

Keep in mind that none of this affects the FlexSystems Manager that will be used with the newly announced IBM PureFlex systems. I'll get into that topic soon.

So what are your thoughts about this change? Had you done much with the SDMC, or were you taking a wait and see attitude?

A Good Look at PowerHA

Another great Virtual User Group webinar recently took place, this one featuring Shawn Bodily's presentation, "Introduction to PowerHA SystemMirror for AIX Standard Edition." Be sure to get the presentation materials and listen to the replay. And look forward to the next VUG webinar, when this topic will be continued.

This material is very similar to the presentation on the same topic at last fall's IBM Technical University. That's an indication of the interest surrounding this solution. As Shawn notes in his presentation, IBM has unveiled 23 major releases of the product, an average one release per year. More than 12,000 customers use it worldwide. While Standard Edition is the focus of Shawn's presentation, he points out that PowerHA SystemMirror for AIX Enterprise edition allows for multi-site cluster management, while also including all the functionality of Standard edition.

Shawn notes that "PowerHA gives you cluster management for the data center. The software monitors, detects and reacts to events. It establishes a heartbeat between the systems, and enables automatic switch-over." He then defines what HA solutions generally try to do, which is eliminate single points of failure. The goal is to reduce downtime, but HA can also help you with planned downtime as well. It might not be fault tolerant, but it will be fault resistant.

One chart shows how failure points can be eliminated. To eliminate node failure, use multiple nodes. To eliminate VIO failover, use dual VIO servers. To eliminate site failure, deploy additional sites. Other items are covered, but again, the idea is to build in redundancy so you can continue to provide access to the applications and data that the business runs on.

Shawn discussed his customer interactions (an area I've touched on previously):

"First, another IBM representative will tell the customer about the hardware and the systems' reliability, availability and serviceability (RAS) features. Then a second rep will discuss live partition mobility and how it seamlessly shifts logical partitions from one frame to another. So after 20-30 minutes of hearing about how the hardware never fails, THEN Shawn must step in and explain why the customer should be concerned with high availability and disaster recovery. That's one tough act to follow."

Some additional slides introduce Live Partition Mobility and explain how it allows you to move your live running operating system and application from one physical frame to another. Of course, that's a hardware maintenance solution. What about software maintenance? With PowerHA, you can fail over to the other node, upgrade your application or your operating system, then fail back and do the same thing on the other side. Shawn notes that basically, PowerHA performs the functions that LPM doesn't. He also gets into how PowerHA is used to recover from node failures, network failures, loss of shared storage access, and -- with version 7.1 based on Cluster Aware AIX technology -- rootvg errors. Finally, Shawn covers the differences between PowerHA 6.1 and 7.1.

Shawn makes these additional points:

* Remember, we can fail over from one system to one system, from one system to any other system, from any system to one, and from any to any. We can also fail over between different versions of hardware from the same or different families, assuming you can live with the performance degradation, and after you verify that the version of the operating system you want to run will work with your particular configuration. Failing POWER7 to POWER6 or POWER5 could conceivably work as long as you verify that your particular setup is supported.

* Often you'll want to make your service IP address highly available, along with your application server and your shared storage.

* You can create user defined resources, custom resource groups and more granular resource group options. You can set up resource group dependencies. In his example, you might want to be sure your database is running before you try to start application servers, so you could configure that as a dependency.

* You can configure different priorities and choose which nodes to your resource starts on. This provides a great deal of flexibility and control when setting things up.

* You can also configure things to automatically run DLPAR and COD operations. So you could have a very "skinny" standby node, but when needed it could perform operations on the HMC and bring additional memory and CPU resources online.

* You can have application monitors so you can take actions if PowerHA detects that the application has gone down, and you can set up file collections to have the software help you keep config files or other important files kept in sync across the cluster. This is meant to support all regular files, as opposed to things like trying to keep password files in sync.

* Configuration and smart assistants are available to configure clusters. System Director plugins are also available, both for managing and monitoring your cluster's state.

* The CAA command -- /usr/sbin/clcmd -- can be used to distribute commands across all cluster nodes.

* A cluster test tool can be used to validate clusters. This is also a good way to run tests across many different clusters in the environment, to ensure that we're running the same tests across all of the machines.

Shawn's final slide lists these great resources:

* IBM Redbook: PowerHA SystemMirror 7.1 for AIX

* PowerHA & Aix Support & Compatibility Matrix

* PowerHA Hardware Support Matrix

Incidentally, you can follow Shawn on Twitter.

I really enjoy these webinars, and of course the availability of the replays and presentation materials is a huge convenience. Hopefully my writing about VUG webinars encourages you to take the time and listen for yourself.

Note: Completely unrelatedly, I'm a YouTube sensation. Well, maybe not, but I'm there. Five new videos just went live where you can see me and some other IBM Power Champions talking with IBM's Ian Jarman at last fall's Technical University.

 

 

 

IVM, HMC and SDMC Continued

Continuing from last week, here's more on the recently released IBM Redpaper, "IBM PowerVM Getting Started Guide."

Chapter 2: IVM

From the authors:

"IBM developed the Integrated Virtualization Manager (IVM) as a server management solution that performs a subset of the HMC and SDMC features for a single server, avoiding the need for a dedicated HMC or SDMC server. IVM manages a single stand-alone server -- a second server managed by IVM has its own instance of IVM installed. With the subset of HMC and SDMC server functionality, IVM provides a solution that enables the administrator to quickly set up a server. IVM is integrated within the Virtual I/O Server product, which services I/O, memory, and processor virtualization in IBM Power Systems.

"There are many environments that need small partitioned systems, either for test reasons or for specific requirements, for which the HMC and SDMC solutions are not ideal. A sample situation is where there are small partitioned systems that cannot share a common HMC or SDMC because they are in multiple locations.

"IVM is a simplified hardware management solution that inherits most of the HMC features. It manages a single server, avoiding the need for an independent personal computer. It is designed to provide a solution that enables the administrator to reduce system setup time and to make hardware management easier, at a lower cost.

"When not using either the HMC or the SDMC, VIOS takes control of all the hardware resources. There is no need to create a specific partition for the VIOS. When VIOS is installed using the default settings, it installs on the server’s first internal disk controller and onto the first disk on that controller. IVM is part of VIOS and activated when VIOS is installed without an HMC or SDMC."

Chapter 2 continues with details on IVM installation.

I wish this chapter would include screen shots. (There are screen shots in chapters 3-4.) The Redpaper describes the steps, but for those unfamiliar with the interface it might be confusing. Some screen shots could help.

Chapter 3: HMC

More from the authors:

"Note: There is flexibility for you to plan your own adapter numbering scheme. The Maximum virtual adapters setting needs to be set in the Virtual Adapters window to allow for your numbering scheme. The maximum setting is 65535 but the higher the setting, the more memory the managed system reserves to manage the adapters."

They cover the three VIOS installation methods: DVD, via the HMC (using the installios command) and via Network Installation Manager (NIM). One of the notes says:

"Interface en5 is the SEA adapter created in 3 on page 29. Alternatively, an additional virtual adapter may be created for the VIOS remote connection, or another physical adapter may be used (it will need to be cabled) for the TCP/IP remote connection. TCP and UDP port 657 must be open between the HMC and the VIOS. This is a requirement for DLPAR (using RMC protocol)."

I know when I set up shared Ethernet adapters on VIO servers, I like to add an additional virtual Ethernet adapter to put my VIO IP address on. This allows me to perform maintenance on my VIOS and SEA without an outage, as the network traffic goes out my backup SEA on my other VIOS.

Section 3.2 covers setting up dual VIO servers:

"The benefit of a dual VIOS setup is that it promotes Redundancy, Accessibility and Serviceability (RAS). It also offers load balancing capabilities for MPIO and for multi SEA configuration setups. The differences between a single and dual VIOS setup are:

The additional VIOS partition
The additional virtual Ethernet adapter used as the SEA Control Channel adapter per VIOS
Setting the trunk priority on the virtual Ethernet adapters used for bridging to physical adapters in an SEA configuration."

The authors explain how to move from a single VIO SEA to a dual VIO scenario by adding the control channel adapter using this command:

   chdev -dev ent5 -attr ctl_chan=ent6 ha_mode=auto

They also mention that we can run commands on the VIO command line or use cfgassist, which is similar to smitty in AIX.

Section 3.3 covers setting up virtual fibre. The authors argue that virtual SCSI disks be used for rootvg and NPIV be used for data LUNs:

"Virtual Fibre Channel allows disks to be assigned directly to the client partitions from the SAN storage system. With virtual SCSI, the disks are assigned to the VIOS partition before they are mapped to a virtual SCSI adapter.

"The preference is to still use virtual SCSI for client partition operating system disk, and use virtual Fibre Channel for the data. The reasons for using virtual SCSI for client partition operating system disks are:

* When the disks are assigned VIOS first, they can be checked before having them mapped to a client. Whereas using virtual Fibre Channel this cannot be determined until the client partition is loaded from an installation source.

* Operating systems such as AIX and Linux have their kernels running in memory. If serious SAN issues are being experienced, the VIOS will first detect the problem and sever the link to the client partition. The client partition will halt abruptly reducing any risk to data corruption. With operating systems using virtual Fibre Channel or physical Fibre Channel, the partition will remain running for a period. During that period the client partition is susceptible to data corruption.

* Operating system disks using virtual SCSI are not reliant on external device drivers whereas operating system disks using virtual Fibre Channel are. When it comes to upgrading the external device drivers, the client partitions would need to follow special procedures to upgrade."

Chapter 4: SDMC

From the authors:

"The IBM Systems Director Management Console (SDMC) provides system administrators the ability to manage IBM Power System servers as well as IBM Power Blade servers. The SDMC organizes tasks in a single panel that simplifies views of systems and day-to-tay tasks. The SDMC is also designed to be integrated into the administrative framework of IBM Systems Director.

"The SDMC can automatically handle the slot allocation of virtual adapters for the user. With the SDMC the user can choose to either let the SDMC manage the slot allocations, or use the traditional manual mechanism to allocate the virtual adapter IDs."

According to section 4.1.4, setting up an SEA failover configuration is a simple GUI operation when using SDMC:

 

 

Select the primary VIOS, the physical adapter you want to use and the backup VIO and its physical adapter. Then hit OK:

"The SDMC automatically creates the SEA adapters on both VIOS1 and VIOS2. The SDMC will also configure the control channel as a part of this step. The virtual Ethernet adapter with the highest VLAN ID is used for the SEA control channel."

This should remove the possibility of errors arising from setting up the control channels manually.

Although you can do the same with the HMC GUI, I still prefer to manage things on the command line.

The publication has much more. It's well worth your time.

One Guide to the IVM, HMC and SDMC

The just-published IBM Redpaper, "IBM PowerVM Getting Started Guide," shows you how to use the Integrated Virtualization Manager (IVM), Hardware Management Console (HMC) and the Systems Director Management Console (SDMC) to configure your systems. It's an extremely valuable guide that's brief enough, at 104 pages, to be read quickly.

The chapters are independent, so they can be read in any order. I'll run down some highlights in posts over the next two weeks:

Chapter 1

* There's a great chart on page 2 that compares and contrasts the advantages and disadvantages for the IVM, HMC and SDMC. 

* Section 1.2 covers planning:

"Be sure to check system firmware levels on your power server and HMC or SDMC before you start. Decide if you will use Logical Volume Mirroring (LVM) -- in AIX LPARs -- or Multipath IO (MPIO) at the VIOS level. Obviously if you are running NPIV you would want to run MPIO at the AIX level. The examples in this paper use MPIO. Make sure your Fibre Channel switches and adapters are N_Port ID Virtualization (NPIV) capable if you will be using NPIV.

Make sure your network is properly configured.

Check the firewall rules on the HMC or SDMC.

Plan how much processor and memory you will assign to the VIOS for best performance."

* The authors recommend using a dual VIOS architecture -- two VIO servers -- to provide serviceability and scalability. So do I.

* Part of planning includes establishing a VIO slot number scheme. While the SDMC automates slot allocation, the authors illustrate their preferred scheme in Figure 1-2 on page 5.

The authors suggest a VIO slot numbering scheme where the server slot is 101, 102, 103, etc. in both VIO servers, and the client is 11, 12 connecting to VIO1, and 21 and 22 connecting to VIO2. When mapped, VIO1 would map 11 to 101, 12 to 102, and VIO2 would map 21 to 101, 22 to 102. I prefer a numbering scheme where my even-numbered adapters come from one VIOS (VIO1) and my odd-numbered adapters come from the other (VIO2), with both client and server using the same numbers. In my case I like 100, 110, 120, 130 coming from VIO1, and 101 111 121 131 coming from VIO2. Of course, you may have your own numbering scheme -- which I'd love to hear about in Comments.

* Section 1.3 covers the terminology differences between Power- and x86-based systems, which can be handy for someone with little or no background managing power systems. This can help them make the transition in terminology between the two.

* Section 1.4 lists some prerequisites for setting up the machines:

"Check that:

• Your HMC or SDMC (the hardware or the virtual appliance) is configured, up, and running.

• Your HMC or SDMC is connected to the new server's HMC port. We suggest either a private network or a direct cable connection.

• The TCP port 657 is open between the HMC/SDMC and the Virtual Server in order to enable Dynamic Logical Partition functionality.

•    You have IP addresses properly assigned for the HMC, and SDMC.

•    The Power Server is ready to power on.

All your equipment is connected to 802.3ad capable network switches with link aggregation enabled. Refer to the Chapter 5: Advanced Configuration on page 75 for more details.

Fibre Channel fabrics are redundant. Refer to Chapter 5: Advanced Configuration on page 75 for more details.

Ethernet network switches are redundant.

SAN storage for virtual servers (logical partitions) is ready to be provisioned."

Chapter 5

 

The next three chapters are devoted to the specific approaches you might choose to take. Chapter 2 covers the IVM, Chapter 3 the HMC and Chapter 4 the SDMC. I'll dissect those options next week. For now I'll briefly discuss Chapter 5 (Advanced Configuration):

"This chapter describes additional configurations to a dual Virtual I/O Server (VIOS) setup and highlights other advanced configuration practices. The advanced setup addresses performance concerns over the single and dual VIOS setup.

This chapter includes the following sections:

• Adapter ID numbering scheme
• Partition numbering
• VIOS partition and system redundancy
• Advanced VIOS network setup
• Advanced storage connectivity
• Shared processor pools
• Live Partition Mobility
• Active Memory Sharing
• Active Memory Deduplication
• Shared storage pools"

* Table 5-1 illustrates an example of virtual SCSI adapter ID allocations.

* Section 5.4 covers advanced VIOS network setup, including link aggregation and VLAN tagging:

"The VIOS partition is not restricted to only one SEA adapter. It can host multiple SEA adapters where:

A company security policy may advise a separation of VLANs so that one SEA adapter will host secure networks and another SEA adapter will host unsecure networks.

A company may advise a separation of production, testing, and development networks connecting to specific SEA adapter configurations.

"There are considerations regarding the use of IEEE 802.3ad Link Aggregation, 802.1Q VLAN

tagging, and SEA:

There is a maximum of 8 active ports and 8 standby ports in an 802.3ad Link Aggregation device.

Each of the links in a 803.3ad Link Aggregation device should have their speeds set to a common speed setting. For example, set all links to 1g/Full duplex.

A virtual Ethernet adapter is capable of supporting up to 20 VLANS (including the Port Virtual LAN ID - PVID).

A maximum of 16 virtual Ethernet adapters with 20 VLANS assigned to each adapter can be associated to an SEA adapter.

A maximum of 256 virtual Ethernet adapters can be assigned to a single virtual server, including the VIOS partitions.

The IEEE 802.1Q standard supports a maximum of 4096 VLANS. SEA failover is not supported in IVM as it only supports a single VIOS partition."

Whether you set up IBM Power Systems all the time or you're just getting started with the platform, this Redpaper is an excellent resource for learning or reviewing the relevant technology and terminology.

Important HMC Fix

This information has been circulating for awhile, and Anthony English covers the topic here and here. But I want to make sure HMC users are aware of this important update and the need to make sure you have the fix loaded if you're at V7R7.3.0.

A problem is known to exist when using dual HMCs in one of two environments: either one HMC is at a different level than the other, or both HMCs are at the base HMC V7R7.3.0 level without fixes.

The problem is possible exposure to corruption that could cause you to lose partition profiles.

A fix is available and should be installed immediately on any HMC that might possibly be impacted by this problem.

If you're using an HMC and an SDMC, be sure to get the fix for the SDMC as well.

From the IBM technical bulletin:

"This PTF was released July 18, 2011, to correct an issue that may result in partition configuration and partition activation profiles becoming unusable. This is more likely to occur on HMCs that are managing multiple systems. A symptom of this problem is the system may display Recovery and some or all profiles for partitions will disappear. If you are already running HMC V7R7.3.x, IBM strongly recommends installing PTF MH01263 to avoid this issue. If you are planning to upgrade your HMC to the V7R7.3.x code level, IBM strongly recommends that you install PTF MH01263 during the same maintenance window to avoid this issue."

The efix can be found here. This package includes these fixes:

• Fixed a problem where managed systems lose profiles and profiles get corrupted resulting in Recovery state which prevent the ability to do DLPAR/LPM.
• Fixed a security vulnerability with the HMC help content.

As noted, this is the statement IBM released in July, before the fix became available. The fix--MH1263 PTF--is now out, so be sure to install it.

Again, from IBM:

"Abstract: HMC / SDMC Save Corruption Exposure
Systems Affected: All 7042s
Communicable to Clients: Yes

"Description:
IBM has learned that HMCs running V7R7.3.0 or SDMC running V6R7.3.0 could potentially be exposed to save area corruption (where partition profile data is stored).

"Symptoms include loss of profiles and/or recovery state due to a checksum failure against the     profiles in the save area. In addition, shared processor pools names can be affected (processor pool number and configuration are not lost), system profiles lost, virtual ethernet MAC address base may change causing next partition activation to fail or to have different virtual Ethernet MAC addresses, loss of a default profile for all or some of the partitions.

"Partitions will continue to run, but reactivation via profile will fail if the profile is missing or corrupted. All mobility operations and some DLPAR operations will fail if a partition has missing or corrupted profiles.

"Environments using HMCs or SDMCs to control multiple managed systems have the greatest exposure. Triggers for exposure include any of the following operations performed in parallel to any managed system: Live Partition Mobility (LPM), Dynamic LPAR (DLPAR), profile changes, partition activation, rebuild of the managed system, rebooting with multiple servers attached, disconnecting or reconnecting a server, hibernate or resume, or establishing a new RMC connection.

"Recommended Service Actions:
Prevention/Workaround:
There is no real work-around other than limiting the configurations to a single HMC managing a single managed system.

"Customers who have not yet upgraded or installed HMC 7.7.3 should delay the upgrade/install if at all possible until a fix is available.

"Customers who have not yet installed and deployed SDMC 6.7.3.0 should avoid discovering     production servers until a fix is available.

"Customers that have 7.7.3 or SDMC 6.7.3.0 deployed should:

• Immediately do a profile backup operation for all managed servers:

    bkprofdata -m <managed system name> -f <filename>

•  Minimize the risk of encountering the problem by using only a single HMC or SDMC to  manage a single server via the following options:

1. Power off dual HMC/SDMC or remove the connection from any dual HMC/SDMC.
2. Use one HMC per server (remove/add connections as needed if necessary).
3. A single HMC/SDMC managing multiple servers might be done relatively safely if the operations listed under triggers above are NOT done to two different servers concurrently.

"Recovery:
 NOTE: Recovery will be easiest with a valid backup of the profile data. So it is extremely important to backup profile data prior to an HMC upgrade or after any configuration changes to the save area. If a    profile data backup exists this problem can be rectified by restoring using:

    rstprofdata -m <managedsysname> -l 3 -f <backupfilename>

"In addition to user backups, profile backups can be extracted from the previous save upgrade data (DVD or disk); a backup console data (if available); or pedbg.

"If a good backup does not exist, call your HMC/SDMC support to determine if recovery is possible.

 "Fix:
A fix to prevent this from occurring is due out by the end of July (Editor's note: We realize this is now available but wanted to include the verbiage for completeness), but the PTF will not fix an already corrupted save area. A follow-up notification will be sent as soon as it is available.”

Please heed the warnings and load this fix as soon as possible if you're running V7R7.3.0. And don't run any HMCs at V7R7.3.0 while running others at a lower level.

Connecting to a Remote HMC

What do the best practices documents tell us about HMC private networks when communicating between HMC and flexible service processor (FSPs)? Is a private network switch or VLAN really needed between the HMC and the FSP? Can an HMC in a remote data center be used to manage machines over regular network links?

This 2007 document, authored by IBMers Ron Barker, Minh Nguyen, and Shamsundar Ashok, states:

"The network connection between the HMC and the FSP can be either private or open on low-end to mid-range servers. Private is preferred, and therefore a best practice. A private network is required for systems that have a BPA, such as the models 590, 595 and 575.

"In an open configuration, the FSP’s IP addresses must be set manually on each managed server. They cannot be DHCP clients of any server other than a managing HMC.

"Addresses can be set using the Advanced System Management Interface (ASMI) on the FSP. This involves directly connecting a laptop to one of the ports on the FSP and using HTTPS to log into one of the two pre-defined IP addresses. The HMC1 port defaults to 192.168.2.147; HMC2 defaults to 192.168.3.147. The systems administrator can login as user 'admin' using the default password 'admin,' which should be changed during the initial installation for security reasons. If no laptop is available, an ASCII terminal can be used on the native serial port to access the FSP menus in character mode.

"Remember, with POWER7, the addresses have changed to:

Service processor A HMC1 169.254.2.147,  HMC2 169.254.3.147 and
Service processor B (if installed) HMC1 169.254.2.146  HMC2 169.254.3.146

"Open networks are used for communications between a logical partition and the HMC. This connection is largely to facilitate traffic over the Resource Monitoring and Control (RMC) subsystem, which is the backbone of Service Focal Point (SFP) and required for dynamic resource allocation. The open network also is the means by which remote workstations may access the HMC, and it could be the path by which an HMC communicates with IBM Service through an Internet connection."

Though HMCs will be going away in favor of SDMC, the transition will be gradual. For the time being we need to keep our HMC skills sharp, and this is one question that frequently arises when customers plan to add new systems to an environment.

Recently I helped a client set up HMC communications over an open network in just this manner. We found that our FSP ports were still going 100MB, as the ports were requested to be set to 1000/Full Duplex. They wouldn't link up to the network at that speed.

Once we had the HMC on the open network, we pointed the HMC to the new IP addresses we'd configured on the FSPs on the local machines. That worked. We then added managed systems that were located in a remote data center without a problem. Finally, we did the same thing with an HMC from the remote data center to manage the machines in the local data center.

While I wouldn't recommend using a remote HMC for day-to-day tasks if a local one is available to you, this is a viable option when setting up machines.

How is the HMC set up in your environment? Please share your experiences in Comments.

Setting Up NPIV

Following up on this recent post, I want to go into greater detail on setting up NPIV (N_port ID virtualization).

With most customers, the first question I get is, "Do I have the hardware to run NPIV?" If you're running at least POWER6, you have IBM 8-GB fibre cards and your SAN switches are NPIV-capable, you should have what you need.

This document can help you determine if you're set to use NPIV. If you log into your VIO server, run lsnports and find the value for "fabric" is 1, you'll know you can safely map virtual adapters to your physical adapters. (Also remember to read the configuration document I referenced in the previous post.)

Setup is straight-forward. Create a virtual fibre adapter in your VIO server, then create a virtual fibre adapter in your VIO client. Map the virtual adapter in the VIO server to a physical fibre adapter using
the vfcmap command and give the virtual worldwide name (WWN) to your SAN team.

Lately I've done a number of logical disk migrations for people who initially set up virtual SCSI and want to move to NPIV. Using dynamic LPAR, virtual fibre adapters can be added to the VIO server and client. The virtual adapter is mapped to the physical adapter and WWNs are obtained from the HMC. If you use Live Partition Mobility in your NPIV environment, remember that you'll need to map both virtual WWNs, as both are used during the actual migration.

NPIV allows you some flexibility as far as using virtual adapters. I've seen some environments that have one adapter per VIO server, and others that map a virtual fibre adapter to every physical adapter in their VIO server. Some argue that one virtual adapter per VIO reduces complexity while providing sufficient redundancy. In many of these environments, the first virtual adapter is mapped to fcs0, the second to fcs1, etc. Whichever method you choose, I believe it's important to test the set-up by rebooting the VIO servers. You need to verify that what you think will happen when you bring down a VIO server is what will actually happen.

I have customers that reuse the same LUN that they were using with vSCSI. In those cases, we unmounted the filesystems, varied off and exported the volume groups, used the rmdev command to remove the disk and the disk's mappings from both VIO servers, changed the SAN zoning to map to the virtual WWN instead of the VIO servers' physical WWN, ran cfgmgr in the client LPAR to see the disk directly in the client (importvg –y vgname hdiskX) and mounted the filesystems. It's almost as if we never made any changes -- though you need to be aware of any disk drivers or MPIO software that's now needed in the client instead of the VIO server.

I also have customers that -- rather than go through the downtime associated with remapping their disks – are fortunate enough (because they have enough storage) to just create new LUNs. They leave their original vSCSI mappings in place, map their new LUNs via NPIV directly to the client and just use migratepv to move the data from the old disks to the new disks. Then they remove the old vSCSI disks and mappings at their leisure.

One other thing to keep in mind once you complete the move to NPIV: Just because you no longer use vSCSI for your disks, you should still keep a vSCSI adapter on your VIO server and client for virtual optical devices. I know I still want the capability to use virtual .iso images as I always have.

So are you looking forward to an NPIV migration project? And if you're already up and running, what's your experience been like? Please share your thoughts in Comments.

The SDMC Evolution

The IBM Redbook covering the IBM Service Director Management Console (SDMC) is now available. Whether you're making the move from the HMC to the SDMC now or later, this publication will help you with your transition. It's well worth the download.

The first time I read it, I learned interesting things like:

* "The SDMC is available as a software and a hardware appliance. The software appliance will replace the Integrated Virtualization Manager, and can manage machines from the blades up to the 750 class servers. The hardware appliance is required for management of midrange systems and high-end systems. The SDMC releases can be used alongside the Hardware Management Console during trials and deployment, which eases transition."

* "The SDMC virtual machine contains Linux as the base operating system. For the software appliance, the client supplied virtualization options for different hypervisors include Red Hat Enterprise Virtualization KVM or VMware ESX/ESXi."

* Section 1.5 shows us how the terminology will evolve. Managed systems are now called servers, frames are power units, LPARs are virtual servers, the hscroot ID becomes the sysadmin ID, partition
mobility becomes relocation, etc.

* "The SDMC incorporates most functions of the Hardware Management Console. This has been done through direct mapping of commands or by replacing functions that are present already in IBM Systems Director. Some functions are not available in the first release of the SDMC, notably the ability to handle system plans."

Comment: As system plans are wonderful tools that I highly recommend, hopefully this will be fixed very quickly. From what I understand this will be addressed in one of the early service packs.

* "The command-line interface has been mostly kept the same. On the SDMC, most of the commands are just preceded by smcli. This new prefix might require changes to existing scripts that use the Hardware Management Console."

* "SDMC provides the capability to back up the whole virtual machine onto removable media or a remote FTP server. You can restore using the backup file from the removable media or from a remote FTP server. The restore will be full image deployment and all existing files will be replaced from the backup. Unlike the HMC, SDMC backs up the entire disk instead of individual files. The backup function requires that the SDMC be temporarily shut down to quiesce the disks, but it will be immediately restarted while the disk files are copied to removable media or a remote FTP server. The restore function takes under an hour to complete."

The SDMC has been a topic of discussion at workshops and IBM Technical University conference, so hopefully most customers are up on this change. It shouldn't come as a surprise.

Basically, the SDMC is still an appliance just like the HMC is today. It will run Systems Director code under the covers. The hardware will be the same CR6 that we're used to, but SDMC will require more memory and disk space. There will be two 500 GB disks running in a RAID0 setup, so be sure to backup the SDMC; these disks will not be mirrored. Although I've heard that existing CR6 machines will ultimately be upgradeable, at GA the machines will be net new. So initially, it will probably make sense to run the HMC and SDMC simultaneously until you get used to the SDMC's new capabilities.

Those new capabilities are impressive. The SDMC will be able to manage the whole POWER6 and POWER7 lineup, including blade systems. This is a much nicer alternative than the current solution of using and managing each individual blade via IVM. It's a pain to use the GUI and deal with the frequent timeouts that occur when using the IVM interface. Assuming you have sufficient resources, another thing you'll be able to do with the SDMC that can't be done with IVM is create dual VIO servers on your blades.

Finally, the SDMC will support the capability to run live partition mobility operations between blades and standalone servers and back again. This will give customers greater flexibility as far as purchasing hardware and running workloads. With this forgiving infrastructure you'll be able to move workloads around on the fly, and with dynamic logical partition operations you'll be able to adjust hardware allocations on the fly.

The SDMC transition will not be a big bang change from the HMC, but it will take some time. The rollout, in fact, is expected to take years. As is standard practice for IBM when introducing updated solutions, the HMC will continue to be supported through this transition, but over time advanced virtualization capabilities will increasingly be brought to the SDMC (and not necessarily the HMC). Customers are encouraged to try out the SDMC, make a transition plan, run it alongside the HMC and get used to it.

As noted, there is a strong thread between the two solutions. The SDMC, like the HMC, is an appliance with user management capabilities and a built-in firewall. The network topology is identical on both solutions. As with the HMC, the SDMC won't allow admins root access or the capability install software. Just as larger environments have multiple HMCs now, you'll be able to run multiple SDMCs. You'll still need an additional Systems Director server to manage your SDMC stand-alone devices and take advantage of advanced plugins like Active Energy Manager or VMControl.

SDMC availability is planned for May 13.

So what do you think of this change? When do you expect to see an SDMC in your environment?

IBM Product Preview

IBM is conducting what it calls a "product preview" today. The subject of this preview is new hardware that is expected to be rolled out later this year.

I received this information during a recent conference call with IBM.

First, there will be a new POWER7 high-performance computing machine, the Model 775. As you'd expect, this machine is based on POWER7 processors, which come with eight cores per socket running at 3.8 GHz. The 775 will be packaged with a Quad Core Module (QCM). A QCM consists of four POWER7 chips; thus, each QCM will have 32 cores.

IBM will then take the QCMs and integrate them into what they call a drawer, or a node. A 2U drawer will have eight QCMs, giving us 256 (8x32) total cores in an efficient, densely packed 2U form factor. You'll be able to have 2TB of memory per node, and IBM estimates peak performance of 7.8 teraflops in a 2U package.

The machine will have a high-speed interconnect fabric, which allows these 2U nodes to be connected using an optical interconnect. This will provide the capability to connect to a total of four drawers (IBM calls this a supernode) consisting of 1,024 cores. Twelve nodes will fit in a rack, with a maximum of 24 TB per rack and a peak performance of 95 teraflops in the rack. Optical interconnects allow for connecting supernodes together -- as of now customers could have as many as 512 supernodes (or 524,000 cores) running at an estimated 15 petaflops at peak performance.

The 775 will be a quiet machine, because it will have fewer fans. It will be water-cooled with 100 percent heat capture. The 775 is expected to be used for climate and weather modeling and prediction, life sciences, nuclear resource management, and financial services.

Here are some other things that grabbed my attention:

* The 795 is expected to have new capabilities that allow for hot node add, hot memory upgrades and repair. You'll also see concurrent GX adapter add and hot GX adapter repair, along with concurrent system controller repair. The maximum number of partitions that can be created on a frame will increase to 1,000 on the 795, 640 on the 770 and 780, and 320 on the 750. Relatedly, when active energy manager is used with these machines, administrators will be able to set up energy policy definitions by partition rather than by system, so different policies can enable energy savings while maintaining performance.

* New blades are planned. There will be a new single-wide Model 703 blade with a maximum of 16 cores running at 2.4Ghz and 128 GB of memory. There will also be double-wide Model 704 blades, which consist of 32 cores running at 2.4GHz and a maximum of 256 GB of memory. The 703 is expected to have one hard disk bay, and you can choose either an HDD or an SSD. The 704 will have two disk drive bays, so you can have either two HDDs or four SSDs. The blades would provide the capability to run both traditional rotating hard drives and SSD drives. These new blades are expected to run in BladeCenter H, HT or S chassis.

* The Model 750 will be refreshed with new processor options, including 4-core 3.7 GHz, 6-core 3.7 GHz and 8-core 3.2 and 3.6 GHz options. The 750 will still have four sockets and 512 GB of memory per machine.

* Support for dual VIO servers across all of the POWER7 blades will be enabled through the new Systems Director Management Console (SDMC). SDMC will be used to enable active memory expansion on the blades. There will also be support for running live partition mobility operations between blades and rack servers, which will open up a whole new way to manage workloads. The SDMC will run on familiar CR6 hardware, although with beefier disk and memory requirements.

The SDMC enhancements mean we'll no longer need to run IVM to manage our blades, and since we'll be able to run dual VIO servers, this will make blade offerings a much more attractive option to many customers. In addition, because IBM is making the SDMC the next-generation management console for Power systems, the HMC will be phased out over the next several years. (Although the HMC and IVM are expected to be kept current with new Power systems models into 2013, they will not incorporate future advanced management capabilities.) During the transition period, customers will be able to run the SDMC side by side with existing HMCs until they're ready to switch permanently.

On the call it was stressed that the SDMC is meant to be evolutionary rather than revolutionary. In other words, IBM says it will give customers ample time to make this transition. And really, this shouldn't come as a surprise. For awhile now, I've been hearing that "the HMC is going away" at conferences and workshops I've attended.

The SDMC will manage POWER6 and POWER7 servers, and there will be a virtual appliance version for small-tier systems. The SDMC will utilize the Systems Director user interface. It will support a superset of HMC capabilities, integrate platform and OS management, and maintain compatibility for CLI and scripting support.

I'll write more about this in the near future, but rest assured this solution will make it much easier to manage an entire computing environment -- including servers and blades -- from a central location. In addition, read IBM Systems Magazine for more details about the SDMC (a cover story is planned for the May 2011 issue). And an IBM Redbook on the SDMC is expected to be ready later this month.

* Another interesting option that I saw was an SAS disk-only I/O drawer (the EXP24S) that could house up to 24 SFF drives in a 2U form factor. This I/O drawer would allow you to partition the drawer into four different sets of disks, thus making it easier to present a smaller group of disks in the drawer to different partitions. This could be a nice option if you're not using a SAN but still need access to more external storage.

* Finally, IBM highlighted a change to the Power systems landing page on its website. Look for this URL: www.ibm.com/power.

So what do you think of these planned solutions? I expect there to be plenty of discussion around the HMC and SDMC as we learn more in the coming months.  Please leave your thoughts in Comments.

10 Rules for Admins

A few months ago I took a class with IBMer Tommy Todd, who highlighted 10 rules for administrators that he had accumulated over the years. I'll run down his list, and comment about each rule. Then I'd appreciate your thoughts.

Documentation: Make sure your documentation is up to date. Ask yourself how you're documenting your systems. I really like to generate a sysplan from the HMC. It shows me a diagram of the physical hardware, where the adapters are assigned, how the LPARs are configured, etc.

Make backups: How are you backing up your machine? Do you backup both the operating system (rootvg) and data (datavg)? Are you periodically running mksysb commands, and have you tested them? Can you restore your machines? Have you tested your disaster/recovery plans? Did you back up your HMC and VIO servers?

Try it three times: Did you fat finger something? Do you have poor typing skills? Did you use the wrong flag? Do you need to go look at the man pages?

Don't overlook the obvious: Many times the answer will be simple. Recently someone was trying to remove a directory and couldn't do it. Fuser, lsof -- nothing was showing that the directory was in use. The admin was stumped. It turned out he still had a mounted filesystem on that mountpoint. Once he unmounted the filesystem, he was good to go. How many obvious things have you overlooked?

Try it, it might work: I like to log into test machines and try different things; you never know what you'll learn. For me the best learning is hands-on learning.

Never say never, always avoid always: There will be exceptions, there is usually more than one way to reach the same endpoint. In other words, don't say "it always works that way," or "it will never work like that." The technology does change. Things that didn't work before do now, and vice versa.

Make a copy before you edit anything: You might have a copy out on a TSM server or backed up somewhere, but what if that backup copy has an issue? It's nice to have that safety net, but it's smart to cp /etc/hosts /etc/hosts.orig before making file changes. If you find yourself making changes to /etc/inittab without using chitab, be sure you back it up first.

There's usually another way to do it: Especially in UNIX, there's more than one way to do something. The religious wars come up when people believe that theirs is the only way. I like to hear about how other people do things and learn from them. Many times they do things their way because they had issues in the past. We can all learn from others' mistakes and benefit from their hard-earned knowledge.

Login as yourself, switch to root when it's needed: With tools like sudo and role based access control (RBAC), do we really need to be logging in and moving around as root? One wrong keystroke can spell disaster when you have super-user authority.

Don't say, "I'll go back and fix that later": There's no time like the present to fix your issues. If you must "fix that later," be sure document it somewhere so you have a reminder to actually come back and fix it later.

Never keep your resume on the system you're supporting: What if the machine crashes and you don't have it on a backup server? What will you do then?

Do you abide by all these rules? What are your own rules? Please register your thoughts in Comments.