Implementing a Shared Storage Pool

I wrote about shared storage pools (here and here) back in January. Recently, I had an opportunity to implement one with a customer.

We had two 720 servers, each of which had two VIO servers. We upgraded to the latest VIOS code, making sure our HMC and firmware were at current levels. Then we presented the LUNs from our SAN, following the steps outlined in my January posts.

First we made sure all the LUNs were set to no reserve.

            chdev -dev <device name> -attr reserve_policy=no_reserve

Then we created the cluster. While I'm giving the names referenced in Nigel Griffiths' presentation (see my first post linked above), for the record, we used our own names.

            cluster -create -clustername galaxy -repopvs hdisk2
            -spname atlantic -sppvs hdisk3 hdisk5 -hostname bluevios1.ibm.com

With that accomplished, I could see that we had a cluster.

            cluster –list

            cluster -status -clustername galaxy

In our cluster, we used one 20G repository LUN and two 500G LUNs for our data.

The cluster –create command took a few minutes to run. On our first try we didn't have fully qualified hostnames on our VIO servers, so we got an error when we tried to create the cluster. Changing the names was easy enough, and after that, the cluster was successfully created.

We ran the cluster –list and cluster –status commands again and got the output we expected. Then from the same node we ran the cluster –addnode command to add a second VIOS to our cluster.

            cluster –addnode –clustername galaxy –hostname redvios1.ibm.com

It took about a minute to add that node, and it was successful. We ran cluster –status again to confirm that the second VIOS was added.

One thing I liked about the process is that the output provides the node name, machine type and model information. This way it's easy to see determine which physical server is running the command.

We did the same procedure for the next two VIO servers. This took a bit longer, likely because they were on another physical server. Still, at the end of the procedure the cluster -status command displayed all four VIO servers in the cluster. When we logged into each of the other VIO servers and ran cluster –status, we saw the same output.

(Note: Running lspv won't tell you that the disks in your storage pool are in use, but the lspv -free command will give you this confirmation. This could be an issue if you were mapping the entire hdisk to a client LPAR -- i.e., the "old" way. But because you're not actually mapping hdisks directly, this isn't necessarily a problem.)

To create a new vdisk to map to our client LPAR, we ran:

            mkbdsp -clustername galaxy -sp atlantic 16G-bd vdisk_red6a -vadapter vhost2

Once we had our disk created and mapped, we ran:

            lssp –clustername galaxy –sp atlantic –bd

That showed us that vdisk_red6a was in the cluster.

Then we ran this command to map it in vios2:

mkbdsp -clustername galaxy -sp atlantic -bd vdisk_red6a -vadapter vhost2

If you compare the command that creates the vdisk to the one that maps the vdisk to the client LPAR, the only difference is the size you provide. Someone can tell me if there's an easier way to do it. For my own amusement I tried using the old mkvdev command. It didn't work.

When we ran lsmap –all, we could see the same vdisk presented to the client, going from both VIO servers.

We then wanted to try live partition mobility using shared storage pools. This posed some problems, but searching on the error message we encountered (HSCLA24E) turned up this entry:

"This week we were trying to migrate some VIO hosted LPARs using XIV disk from one POWER7 system to another. The disk is hosted on a VIO server via the fabric, then using VSCSI devices to map up to the servers. Unfortunately the migration failed and the message we got was HSCLA24E: The migrating partition's virtual SCSI adapter 2 cannot be hosted by the existing virtual I/O server (VIOS) partition on the destination managed system. To migrate the partition, set up the necessary VIOS hosts on the destination managed system, then try the operation again.

"So we did some searching and found the following:

"HSCLA24E error:         

1) On the source VIOS partition, do not set the adapter as required and do not select any client partition can connect when you create a virtual SCSI adapter (can cause the error code).
2) Max transfer size of the used hdisk may not be different on source and destination VIOS.
3) The VIO servers may not have set the reserve_policy to single_path, no_reserve is required.
4) Destination VIO servers are not able to see the disks the client needs.
5) The same VTD (virtual target devices) names may not exist on the destination system."

In our case we addressed no. 1 by unselecting the "any client can connect" option and mapping to the specific client we were using. With these changes, we could successfully migrate the LPAR.

In the course of changing the adapters, we rebooted the VIO servers. Be patient when rebooting. It seems to take some time for the servers to restart and join the cluster. You'll know it's ready when the cluster –status command changes from "state down" to "state OK." (We joked that you only have to give it until "despair + 1.")

Also, be sure to run df and check your /var/vio/SSP/'clustername' filesystem that gets created on all the members of your cluster. That was a quick and dirty way for us to determine that our status was about to change to OK. As the cluster comes online, and as you run cluster –status, you'll see the filesystems mount and the status change from down to OK.

This initial build-out of shared storage pools offers some advantages. For starters, there are fewer, larger LUNs to present and zone to the VIO servers. With larger LUNs being carved up in the pool, there's less to manage with no reserves and the mkvdev commands. Of course some would argue that this advantage is offset by the need to run mkbpsp commands on both VIO servers.

It's also nice being able to login to one cluster node, create a vdisk and see that new vdisk show up on all four nodes, rather than having to login to each VIO server separately. This just feels like a cleaner disk management solution.

As I continue to work with shared storage pools, I'm sure I'll have more lessons to pass along. If you've been using this technology, please share your thoughts in Comments.

The Connection to Storage

In case you're wondering why this server blog just published a post about storage, it's simple: Without storage, servers don't have anywhere to read and write their data. Many of us server admins do have some knowledge of storage, but many more do not. Understanding the differences between storage technologies is important. It can help us when we need to discuss our options with our storage friends.

Anyway, back to Norman Bogard's storage webinar. By the way, although I wish everyone had access to this training, it was provided exclusively for IBMers and IBM Business Partners. The content is not available online, which is why I'm posting this information (with Norman's permission).

I'll continue with a rough summary of how he compares and contrasts network-attached storage and storage area networks. We'll start with SANs:

"Block-level storage devices and SANs like IBM V7000, IBM DS8000 and IBM XIV provide access to equal sized blocks of storage, and the blocks are found by block numbers on a device. All read and write operations are performed on data blocks – mainly using the SCSI protocol. Block services are segmented into LUNs or vdisks, and you might usually have a few dozen of them.

"On the other hand, NAS devices like N series, SONAS or V7000 Unified provide access to files.  These files are found by a name within a tree of names: read, write, create, delete and many more. CIFS, NFS, FTP and other protocols are used to access these files. Device services are exposed as exports, directories and files, and in this case we might be accessing a few hundred, or possibly even millions or billions of files.

"Your NAS may be connected to hundreds or thousands of client machines. Authorization is handled by user IDs for reads, writes and meta-data operations.

"Who owns the filesystem? With direct-attached storage, it's a simple case. The storage lives on the server. Think of regular SCSI disks and expansion drawers filled with disks, or old SSA drawers of disks.

"With a SAN, the server still owns the files and it controls how the data is written to the disk, even though those disk arrays might be on a disk subsystem instead of internal disk. NAS, on the other hand, handles the filesystems and the files and just gives you access to them after you have authenticated.

"With converged, or unified, storage, there are two fundamental approaches to intermixing block and file storage within a single system. IBM’s N series uses block on file. A device file with a logical unit number (LUN) assigned to it is stored within the file server’s WAFL (write anywhere file layout) file system and then mapped to a host. File and block data are stored within the same file system.

"IBM’s Storwize V7000 Unified uses file on block instead of WAFL. A raw device from the V7000 is mapped to hosts. File data is contained within discrete devices. Host block data is contained within discrete devices. File and block data are stored independently.

"Based on your application type, rules of thumb can help you decide whether NAS or SAN makes more sense in your environment.

"Applications and data types that typically reside in block stores or SAN include RDBMS (Oracle, SQL Server, DB2), analytics (stream processing), OLTP, metadata layers (component of content management), e-mail (MS Exchange, Lotus Notes) and virtualization stacks (VMware: VDI, VMDK implementations, HyperV; Citrix Xen)

"Applications/data types that typically reside in files or NAS includes rich media (pictures, videos, seismic data, medical imaging, etc.), VOD, AOD, IPTV analytics (SAS grid), enterprise content management (ECM, e.g., web stores), research data sets, user files (documents, etc.), product lifecycle/data management (PLM/PDM) and virtualized environments (VMware client-driven deployment).

"Another consideration is how you backup these different environments. With NAS you typically have consistent snapshots since the files and filesystems are consistent on the NAS device. Replication is supported, and NAS usually integrates with backup software.

"With a SAN, integration with the host file system is needed to ensure consistency. Many times backups are moved through a master media server to disk or tape. Replication is supported once the file system is consistent."

The webinar also covers file systems, file shares, network services, authentication and authorization, quota, data availability, data protection using snapshots, backups and replication, antivirus support and file cloning.

I love education in this format. This webinar takes a big concept like storage and breaks it down to easy to comprehend descriptions. We may work on servers, but servers connect to storage. That's why learning about storage is worth our effort.

Valuable Insight into Storage

I recently took some online training that I found interesting and valuable. The webinar presenter, Norman Bogard, compares and contrasts network-attached storage (NAS) and storage area networks (SANs). (In his presentation Norman acknowledges Brett Cooper and Nils Haustein for their input, so I want to be sure to mention them here.)

When I think of early iterations of direct-attached storage, like SSA disks or regular internal SCSI disks, it amazes me how far we've come. Advances have been brought not just to storage hardware, but to the network infrastucture. Networks and switches are so much faster and more robust now.

The webinar opens with a review of the history of NAS and SAN and protocols like NFS, NCP, SMB and CIFS. Then some terminology is introduced. I'm paraphrasing and borrowing some of the language from the slide deck since Norman did such a good job of compiling the material:

"SAN, or block storage, will leverage Small Computer System Interface (SCSI) commands to read-write specific blocks. Common SCSI access methods include Fiber Channel (FC), Internet Small Computer System Interface (iSCSI), or InfiniBand (IB). InfiniBand is a high speed network interconnect.

"NAS, or file storage, reads and writes files instead of blocks. The NAS has control of the files, contrasted with a SAN where the server would have control over the files.  A file server is a storage server dedicated (primarily) to serving file-based workloads.

"A NAS gateway is a server that provides network-based storage virtualization. It provides protocol translation from host-based CIFS/NFS to Storage Area Network (SAN) based block storage. Examples of NAS gateways are IBM N series & SONAS; NetApp V Series; EMC VNX/Celerra; OnStor (LSI); HP P4000 Unified Gateway.

"Unified Storage is a single logical, centrally managed storage platform that serves both block (FC, iSCSI, IB) and file-based (CIFS, NFS, HTTP, etc.) workloads. Examples of Unified Storage includes IBM N series; NetApp V series; IBM Storwize V7000 Unified.

"When you compare NAS and SAN, you will find that they have similar concepts. For example, your redundancy for your SAN will come from your MPIO or SDD drivers, while redundancy for the NAS will come from teaming or trunking your network ports for resiliency or improved bandwidth, depending on how you have set things up.

"Your security for a SAN will come from LUN Masking and Zoning, while you would control access on the network the same way you always would, with things like VLANS, exports, and shares.

"Your physical connectivity to the SAN would come via the HBA, while your network traffic for the NAS would go out the same NIC that it always has, at least until converged network adapters become more widely deployed. Once we have converged adapters, all of the traffic will be network traffic, although you will then be dealing with more encapsulation of the different frames and protocols.

"Your underlying protocol on a SAN is SCSI, while you use the same IP/UDP protocols that you do with networking when you use NAS.

"You will call your SAN devices arrays, and you will present LUNs, while your NAS will have filers and data movers. You will have structured/relational data on a SAN, and unstructured data on a NAS."

From there, Norman contrasts the concepts of block storage (SAN) and file storage (NAS). I'll share more about this webinar in next week's post.

 

Automatically Changing IP Addresses in a D/R Environment

I recently spoke to a customer that has its primary and backup servers in different locations. The customer boots from a SAN, with the SAN replicating from site 1 to site 2. In the event of a disaster, the customer wants to fire up its site 2 LPAR from the replicated copy of rootvg. However, the networks are also in different locations.

Rather than do some admin kung fu to allow each network to have the same IP address when it boots up, the customer sought the capability to easily change the IP address depending on the frame being used to boot the LPAR. The customer says this functionality is available in VMware's recovery management product, and wanted to know if the same type of thing can be done from the HMC.

I checked with a couple of my IBM contacts to see if they had any ideas. Chris Gibson had a good one.

"You could write a script that lives on the source system. It checks the system ID (lsattr -El sys0 -a systemid) when the LPAR boots. And if it's a particular system serial number, it could bring up the
interface with a different IP address."

I forwarded this suggestion to the customer, and literally within a day their script was working. With the customer's permission I'm sharing it here, along with their caveat:

"It works … the script is pretty rough. I'm no shell script expert by any means, but it does what I need it to do. I have it in /etc/inittab right before the rctcpip stuff, and that seems to work fine."

Before trying to use this script, make sure your domain, nameserver, gateway and primary and backup server information are accurate for your environment. Of course you might be able to simplify or improve what's here, but this should help you get started. Also note that in the process of posting this script to the blog, some of the formatting may be altered. You savvy scripters should move things around if need be.

#!/bin/ksh
# This script checks to see whether the system is booting off hardware
#at the primary or backup site and sets the IP, gateway, and name
#server based on what hardware it is booting from

# check to see which hardware is booting
OPTION=`lsattr -El sys0 -a systemid -F value`
IPADDRESS=`lsattr -El en0 -a netaddr -F value`
HOST=$(hostname)
DOMAIN="mydomain.com"
PRIMARY="IBM,123"
BACKUP="IBM,456"
NAMESERVER="10.9.0.1"
GATEWAY="10.9.16.1"
# set the primary and backup IP to the correct subnet (xx.1 for primary, xx.2 for backup)
PRIMARYIP=`echo $IPADDRESS | awk -v pos=4 -v repl=1 '{print
substr($0,1,pos-1) repl substr($0,pos+1)}'`
BACKUPIP=`echo $IPADDRESS | awk -v pos=4 -v repl=2 '{print
substr($0,1,pos-1) repl substr($0,pos+1)}'`
BACKUPGW="10.9.16.1"
BACKUPNS="10.9.30.32"

echo "Host Hardware: $OPTION"
echo "Current IP: $IPADDRESS"
echo "Primary IP: $PRIMARYIP"
echo "Backup IP: $BACKUPIP"

if [ "$OPTION" = "$PRIMARY"  ]
then
        echo "Running from primary site"
        if [ "$IPADDRESS" = "$BACKUPIP" ] ; then
                echo "Setting IP for primary location"
                /usr/sbin/mktcpip -h $HOST -a $PRIMARYIP -m
255.255.255.0 -i en0 -n $NAMESERVER -d $DOMAIN -g $GATEWAY -A no -t
N/A
        fi
fi
if [ "$OPTION" = "$BACKUP"  ]
then
        echo "Running from backup site"
    if [ "$IPADDRESS" = "$PRIMARYIP" ] ; then
        /usr/sbin/mktcpip -h $HOST -a $BACKUPIP -m 255.255.255.0 -i en0 -n
$BACKUPNS -d $DOMAIN -g $BACKUPGW -A no -t N/A
    fi
fi

As always, I love getting reader questions and submissions. That includes scripts. Please send me your scripts or any other useful tips. We all benefit when you share your expertise.

 

 

IVM, HMC and SDMC Continued

Continuing from last week, here's more on the recently released IBM Redpaper, "IBM PowerVM Getting Started Guide."

Chapter 2: IVM

From the authors:

"IBM developed the Integrated Virtualization Manager (IVM) as a server management solution that performs a subset of the HMC and SDMC features for a single server, avoiding the need for a dedicated HMC or SDMC server. IVM manages a single stand-alone server -- a second server managed by IVM has its own instance of IVM installed. With the subset of HMC and SDMC server functionality, IVM provides a solution that enables the administrator to quickly set up a server. IVM is integrated within the Virtual I/O Server product, which services I/O, memory, and processor virtualization in IBM Power Systems.

"There are many environments that need small partitioned systems, either for test reasons or for specific requirements, for which the HMC and SDMC solutions are not ideal. A sample situation is where there are small partitioned systems that cannot share a common HMC or SDMC because they are in multiple locations.

"IVM is a simplified hardware management solution that inherits most of the HMC features. It manages a single server, avoiding the need for an independent personal computer. It is designed to provide a solution that enables the administrator to reduce system setup time and to make hardware management easier, at a lower cost.

"When not using either the HMC or the SDMC, VIOS takes control of all the hardware resources. There is no need to create a specific partition for the VIOS. When VIOS is installed using the default settings, it installs on the server’s first internal disk controller and onto the first disk on that controller. IVM is part of VIOS and activated when VIOS is installed without an HMC or SDMC."

Chapter 2 continues with details on IVM installation.

I wish this chapter would include screen shots. (There are screen shots in chapters 3-4.) The Redpaper describes the steps, but for those unfamiliar with the interface it might be confusing. Some screen shots could help.

Chapter 3: HMC

More from the authors:

"Note: There is flexibility for you to plan your own adapter numbering scheme. The Maximum virtual adapters setting needs to be set in the Virtual Adapters window to allow for your numbering scheme. The maximum setting is 65535 but the higher the setting, the more memory the managed system reserves to manage the adapters."

They cover the three VIOS installation methods: DVD, via the HMC (using the installios command) and via Network Installation Manager (NIM). One of the notes says:

"Interface en5 is the SEA adapter created in 3 on page 29. Alternatively, an additional virtual adapter may be created for the VIOS remote connection, or another physical adapter may be used (it will need to be cabled) for the TCP/IP remote connection. TCP and UDP port 657 must be open between the HMC and the VIOS. This is a requirement for DLPAR (using RMC protocol)."

I know when I set up shared Ethernet adapters on VIO servers, I like to add an additional virtual Ethernet adapter to put my VIO IP address on. This allows me to perform maintenance on my VIOS and SEA without an outage, as the network traffic goes out my backup SEA on my other VIOS.

Section 3.2 covers setting up dual VIO servers:

"The benefit of a dual VIOS setup is that it promotes Redundancy, Accessibility and Serviceability (RAS). It also offers load balancing capabilities for MPIO and for multi SEA configuration setups. The differences between a single and dual VIOS setup are:

The additional VIOS partition
The additional virtual Ethernet adapter used as the SEA Control Channel adapter per VIOS
Setting the trunk priority on the virtual Ethernet adapters used for bridging to physical adapters in an SEA configuration."

The authors explain how to move from a single VIO SEA to a dual VIO scenario by adding the control channel adapter using this command:

   chdev -dev ent5 -attr ctl_chan=ent6 ha_mode=auto

They also mention that we can run commands on the VIO command line or use cfgassist, which is similar to smitty in AIX.

Section 3.3 covers setting up virtual fibre. The authors argue that virtual SCSI disks be used for rootvg and NPIV be used for data LUNs:

"Virtual Fibre Channel allows disks to be assigned directly to the client partitions from the SAN storage system. With virtual SCSI, the disks are assigned to the VIOS partition before they are mapped to a virtual SCSI adapter.

"The preference is to still use virtual SCSI for client partition operating system disk, and use virtual Fibre Channel for the data. The reasons for using virtual SCSI for client partition operating system disks are:

* When the disks are assigned VIOS first, they can be checked before having them mapped to a client. Whereas using virtual Fibre Channel this cannot be determined until the client partition is loaded from an installation source.

* Operating systems such as AIX and Linux have their kernels running in memory. If serious SAN issues are being experienced, the VIOS will first detect the problem and sever the link to the client partition. The client partition will halt abruptly reducing any risk to data corruption. With operating systems using virtual Fibre Channel or physical Fibre Channel, the partition will remain running for a period. During that period the client partition is susceptible to data corruption.

* Operating system disks using virtual SCSI are not reliant on external device drivers whereas operating system disks using virtual Fibre Channel are. When it comes to upgrading the external device drivers, the client partitions would need to follow special procedures to upgrade."

Chapter 4: SDMC

From the authors:

"The IBM Systems Director Management Console (SDMC) provides system administrators the ability to manage IBM Power System servers as well as IBM Power Blade servers. The SDMC organizes tasks in a single panel that simplifies views of systems and day-to-tay tasks. The SDMC is also designed to be integrated into the administrative framework of IBM Systems Director.

"The SDMC can automatically handle the slot allocation of virtual adapters for the user. With the SDMC the user can choose to either let the SDMC manage the slot allocations, or use the traditional manual mechanism to allocate the virtual adapter IDs."

According to section 4.1.4, setting up an SEA failover configuration is a simple GUI operation when using SDMC:

 

 

Select the primary VIOS, the physical adapter you want to use and the backup VIO and its physical adapter. Then hit OK:

"The SDMC automatically creates the SEA adapters on both VIOS1 and VIOS2. The SDMC will also configure the control channel as a part of this step. The virtual Ethernet adapter with the highest VLAN ID is used for the SEA control channel."

This should remove the possibility of errors arising from setting up the control channels manually.

Although you can do the same with the HMC GUI, I still prefer to manage things on the command line.

The publication has much more. It's well worth your time.

More on Shared Storage Pools

Back in 2010 I wrote about the changes that were coming to VIOS. One of those big changes, shared storage pools, is now a reality. This gives admins another option to consider when setting up disks on Power servers.

In larger companies, disk changes are typically implemented by SAN teams with many other responsibilities and, often, different priorities. However, by allocating storage to the servers up front and setting it up in a storage pool, admins can manage shared storage pools. In doing so, we can be more responsive to requirement changes. And with thin provisioning, we can determine the amount disk we actually use on each server. For the first time since the days of internal disks and expansion drawers, disk is back under our control.

Here's how Nigel Griffiths explains shared storage pools:

"The basic idea behind this technology... is that [VIO servers] across machines can be clustered together and allocate disk blocks from large LUNs assigned to all of them rather than having to do this at the SAN storage level. This uses the vSCSI interface rather than the pass through NPIV method. It also reduces SAN admin required for Live Partition Mobility -- you get the LUN available on all the VIOS and they organise access from there on. It also makes cloning LPARs, disk snapshots and rapid provisioning possible. Plus thin provisioning -- i.e., disk blocks -- are added as and when required, thus saving lots of disk space."

Continuing from last week, here's more from Nigel's presentation.

Since shared storage pools are built on top of cluster-aware AIX, the lscluster command also provides more information, including: lscluster –c  (configuration), lscluster –d  (list all hdisks), lscluster –i  (network interfaces), lscluster –s  (network stats).

In the demo, he also discusses adding disk space and assigning it to client VMs. Keep in mind that you cannot remove a LUN from the pool. You can replace a LUN but you can't remove one.

He also covers thin and thick provisioning using shared storage pools and shows you how to conduct monitoring. Run topas on your VIOS and then enter D (make sure it's upper-case) so you can watch the disk I/O get spread across your disks in 64 MB chunks. From there, Nigel covers how to set up alerts on your disk pool. If you're using thin provisioning, you must ensure you don't run out of space.

Nigel also shares his script, called lspool. It's designed to do the work of multiple scripts by presenting all of the critical information at one time instead of running multiple commands:

# lspool list each cluster and for each list its pools and pool details
~/.profile
clusters=`cluster -list | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo "Cluster list: " $clusters
for clust in $clusters
do
pools=`lssp -clustername $clust | sed '1d' | awk -F " " '{ printf $1 " " }'`
echo Pools in $clust are: $pools
for pool in $pools
do
lssp -clustername $clust | sed '1d' | grep $pool | read p size free
totalLU numLUs junk
let freepc=100*$free/$size
let used=$size-$free
let usedpc=100*$used/$size
echo $pool Pool-Size: $size MB
echo $pool Pool-Free: $free MB Percent Free $freepc
echo $pool Pool-Used: $used MB Percent Used $usedpc
echo $pool Allocated: $totalLU MB for $numLUs Logical Units
alert -list -clustername $clust -spname $pool | sed '1d' | grep $pool
| read p poolid percent
echo $pool Alert-Percent: $percent
if [[ $totalLU > $size ]]
then
let over=$totalLU-$size
echo $pool OverCommitted: yes by $over MB
else
echo $pool OverCommitted: no
fi
done
done

Nigel examines snapshots and cloning with shared storage pools, noting that the different commands -- snapshot –create, snapshot –delete, snapshot –rollback and snapshot –list -- use different syntax. Sometimes it asks for a –spname flag, other times it asks for a –sp flag. Pay attention so you know the flags that are needed with the commands you're running. He also demonstrates how some of this management can be handled using the HMC GUI.

The viosbr command is also covered. I discussed it here.

Nigel recommends that you get started by asking the SAN team to hand over a few TB that you can use for testing. Also make sure your POWER6 and POWER7 servers are at the latest VIOS 2.2 level. It's worth the effort. This technology will save time, boost efficiency and increase your overall responsiveness to users.

Finally, here's Nigel's shared storage pools cheat sheet:

1. chdev -dev <device name> -attr reserve_policy=no_reserve
2. cluster -create -clustername galaxy -repopvs hdisk2
-spname atlantic -sppvs hdisk3 hdisk5 -hostname bluevios1.ibm.com
3. cluster –list
4. cluster -status -clustername galaxy
5. cluster –addnode –clustername galaxy –hostname redvios1.ibm.com
6. cluster -rmnode [-f] -clustername galaxy -hostname redvios1.ibm.com
7. cluster –delete –clustername galaxy
8. lscluster –s or –d or –c or –i = CAA command
9. chsp –add –clustername galaxy -sp atlantic hdisk8 hdisk9
10. chsp -replace -clustername galaxy -sp atlantic -oldpv hdisk4 -newpv hdisk24
11. mkbdsp -clustername galaxy -sp atlantic 16G
-bd vdisk_red6a -vadapter vhost2 [-thick]
12. rmbdsp -clustername galaxy -sp atlantic -bd vdisk_red6a
13. lssp -clustername galaxy -sp atlantic -bd
14. lssp -clustername galaxy
15. alert -set -clustername galaxy –spname atlantic -value 80
16. alert -list -clustername galaxy -spname atlantic
17. errlog –ls
18. snapshot -create name -clustername galaxy -spname atlantic -lu LUs
19. snapshot -delete name -clustername galaxy -spname atlantic -lu LUs
20. snapshot -rollback name -clustername galaxy -spname atlantic -lu LUs
21. snapshot –list -clustername galaxy -spname atlantic
22. viosbr -backup -clustername galaxy -file Daily -frequency daily -numfiles 10
23. viosbr -view -file File -clustername Name …
24. viosbr -restore -clustername Name …
25. lsmap -clustername galaxy –all

Take the time to listen to the replay, and you'll learn even more. I highly recommend it.

Getting Started with Shared Storage Pools

The December AIX Virtual User Group webinar featured Nigel Griffiths' discussion of phase 2 of shared storage pools. If you didn't tune in, download the presentation materials and listen to the replay.

The new shared storage pool functionality is enabled with the latest PowerVM 2.2 service pack, and is a feature of PowerVM Standard and Enterprise. If you already have PowerVM, simply download the VIO server fixpack to obtain these new features. (Note: Because this TL is based on AIX 6.1 TL7, your NIM server must be at AIX 6.1 TL7 or AIX 7.1 TL1 to use your NIM server with your VIO server.)

 One thing to note, as Nigel points out in the presentation, is that the most common VIOS storage options have been around for some time:

1) Logical volumes, created from a volume group and presented to client LPARs
2) Whole local disks
3) SAN LUNs
4) File-backed storage, either from a file system on local disk or a file system on SAN disks
5) NPIV LUNs from SAN

Nigel then discusses the newest option: using SAN LUN disks that are placed into a shared storage pool. This new option, he emphasizes, doesn't eliminate any of the other options. It does not portend the death of NPIV. It's just an additional VIOS storage choice we now have.

Listen to the replay or look over the slides to gather Nigel's thoughts on the benefits of shared storage pools. He explains that fibre channel LUNs and NPIV can be complex. They require knowledge of the SAN switch and the SAN disk subsystem. If you need to make changes, it might take your SAN guys awhile to implement them. This can slow overall responsiveness. That's to say nothing of smaller organizations that don't have dedicated SAN guys. Live partition mobility can be tough work if your disks aren't pre-zoned to the different frames.

With a shared storage pool you pre allocate the disk to the VIO servers. Then it's under your control. You can more easily allocate the space to your virtual machines.

POWER6 and POWER7 servers (including blades) are needed to use shared storage pools. At minimum you should allocate a 1 GB of LUN for your repository and another 1 GB of LUN for data, but in order to be useful, in most cases you'll need much larger LUN(s) – think terabytes of disk -- if you plan to do much with it. 

Your VIOS must have the hostname set correctly to resolve the other hostnames. In Nigel's experience he couldn't use short hostnames -- they had to be changed to their fully qualified names.

He also recommends giving your VIOS a CPU and at least 4 GB of memory. "Skinny" VIOS servers aren't advisable with shared storage pools. Currently, the maximum number of nodes is four, the maximum physical disks in a pool is 256, the maximum virtual disks in a cluster is 1,024, and the number of clients is 40. A pool can have 5 GB to 4 TB of individual disks, and storage pool totals can range from 20 GB to 128 TB. Virtual disk capacity (LU) can be from 1 GB to 4 TB, with only one repository disk.

If you played around with phase one, you'll find that many of your limits have been removed. Now you can use shared storage pools for live partition mobility, perform non-disruptive cluster upgrades and use third party multi-pathing software.

You cannot have active memory sharing paging disks on shared storage pool disks.

Nigel covers the relevant terminology (clusters, pools, logical units, etc.).  He also demonstrates how to actually prepare and set up your disks. In a nutshell you must get your LUNs online and zoned to your VIO servers, and you need to set your reserve policy to no_reserve on your LUNs.

After covering the commands for managing clusters -- cluster –create, cluster –list, cluster –status, cluster –addnode, cluster –rmnode and cluster –delete -- he recommends creating the cluster on one of the VIO servers and then adding additional VIO servers to the shared storage pool cluster. From there, you can allocate space to the VM clients.

Next week I'll have more information from Nigel’s presentation, including scripts and cheat sheets. In the meantime, why not upgrade your test machine's VIO servers to the latest level so you can try this functionality?

Setting Up NPIV

Following up on this recent post, I want to go into greater detail on setting up NPIV (N_port ID virtualization).

With most customers, the first question I get is, "Do I have the hardware to run NPIV?" If you're running at least POWER6, you have IBM 8-GB fibre cards and your SAN switches are NPIV-capable, you should have what you need.

This document can help you determine if you're set to use NPIV. If you log into your VIO server, run lsnports and find the value for "fabric" is 1, you'll know you can safely map virtual adapters to your physical adapters. (Also remember to read the configuration document I referenced in the previous post.)

Setup is straight-forward. Create a virtual fibre adapter in your VIO server, then create a virtual fibre adapter in your VIO client. Map the virtual adapter in the VIO server to a physical fibre adapter using
the vfcmap command and give the virtual worldwide name (WWN) to your SAN team.

Lately I've done a number of logical disk migrations for people who initially set up virtual SCSI and want to move to NPIV. Using dynamic LPAR, virtual fibre adapters can be added to the VIO server and client. The virtual adapter is mapped to the physical adapter and WWNs are obtained from the HMC. If you use Live Partition Mobility in your NPIV environment, remember that you'll need to map both virtual WWNs, as both are used during the actual migration.

NPIV allows you some flexibility as far as using virtual adapters. I've seen some environments that have one adapter per VIO server, and others that map a virtual fibre adapter to every physical adapter in their VIO server. Some argue that one virtual adapter per VIO reduces complexity while providing sufficient redundancy. In many of these environments, the first virtual adapter is mapped to fcs0, the second to fcs1, etc. Whichever method you choose, I believe it's important to test the set-up by rebooting the VIO servers. You need to verify that what you think will happen when you bring down a VIO server is what will actually happen.

I have customers that reuse the same LUN that they were using with vSCSI. In those cases, we unmounted the filesystems, varied off and exported the volume groups, used the rmdev command to remove the disk and the disk's mappings from both VIO servers, changed the SAN zoning to map to the virtual WWN instead of the VIO servers' physical WWN, ran cfgmgr in the client LPAR to see the disk directly in the client (importvg –y vgname hdiskX) and mounted the filesystems. It's almost as if we never made any changes -- though you need to be aware of any disk drivers or MPIO software that's now needed in the client instead of the VIO server.

I also have customers that -- rather than go through the downtime associated with remapping their disks – are fortunate enough (because they have enough storage) to just create new LUNs. They leave their original vSCSI mappings in place, map their new LUNs via NPIV directly to the client and just use migratepv to move the data from the old disks to the new disks. Then they remove the old vSCSI disks and mappings at their leisure.

One other thing to keep in mind once you complete the move to NPIV: Just because you no longer use vSCSI for your disks, you should still keep a vSCSI adapter on your VIO server and client for virtual optical devices. I know I still want the capability to use virtual .iso images as I always have.

So are you looking forward to an NPIV migration project? And if you're already up and running, what's your experience been like? Please share your thoughts in Comments.

Getting Started With NPIV

NPIV isn't new functionality, but plenty of customers are only just now getting started with it. I know this because lately, I'm hearing a lot about NPIV. In response to the numerous queries coming my way, I searched and found this excellent IBM Support document on configuring NPIV:

"N_Port ID Virtualization or NPIV is a Fibre Channel facility allowing multiple N_Port IDs to share a single physical N_Port. This allows multiple Fibre Channel initiators to occupy a single physical port,
easing hardware requirements in Storage Area Network design. An NPIV-capable fibre channel HBA can have multiple N_Port IDs, each with a unique identity and world wide port name."

Compared to using virtual SCSI devices (vSCSI), storage management is greatly simplified with NPIV. NPIV allows AIX admins to zone a LUN to a particular client LPAR directly, rather than use VIOS as a middleman. So with NPIV and SEA, the VIO servers handle the shared Ethernet and NPIV duties. Best of all, there's no need to map and track LUNs -- that duty can be left with the SAN team where it belongs.

In contrast, when using vSCSI with VIO servers, your lsmap –all output can be a mess to manage if a large number of LUNs are being mapped through your VIOS to client LPARs. I've seen servers with hundreds of LUNs being presented to the VIOS. In those cases, the AIX admins must manage the subsets of LUNs that are then mapped to individual VIO clients. All that disk-mapping must be tracked, and I've seen many different spreadsheets and documents that attempt to do this.

In a typical scenario, two VIO servers will be set up (so that one can be serviced or restarted without these activities impacting the client LPARs). A fibre card or two is usually attached to each VIO server. Then the SAN team can zone the VIO servers to the SAN using the World Wide Name (WWN) information from the physical adapters. This results in a pile of LUNs that AIX admins must map to the appropriate VIO clients. To make all of the LUNs accessible from both VIO servers, each LUN's no reserve attribute must be set. So the admins end up doing the mappings twice, once on each VIO server.

On top of that, admins must pay attention to PVIDs or LUN IDs to ensure that the disk that's mapped on VIOS1 is the same one mapped on VIOS2. Having the no reserve attribute set on the disk can open up a potential disaster if the same LUN is accidentally mapped to different clients. If two different clients are booting from the same LUN, it's time to look for a mksysb and do a restore.

One plus with vSCSI is that MPIO software only needs to be loaded on the VIO server. The VIO clients usually just use the built-in AIX MPIO software as they have no visibility to the disks other than recognizing that they're virtual SCSI disks.

From this lengthy explanation on vSCSI, you might have already figured that NPIV, once you have it set up, is much easier to use. And you're correct. With NPIV, virtual WWN information is created for each client LPAR. The SAN team gives LUNs to the client LPARs directly. Virtual fibre adapters must still be mapped to a particular physical fibre card in the VIO server, but admins don't need to map and track LUNs or worry about reserve locks on the LUNs. (We do, however, need to remember to load the MPIO software into the client LPARs, because the clients do recognize the disks and the storage subsystems from which they come.)

I'll have more NPIV info next week, so stay tuned.

New SSD Modules Offer Greater Efficiency

I've been meaning to touch on one other aspect of the recent Power Systems announcements -- that being the new solid-state drive (SSD) disk modules.

The new SSD modules are about the same size as a thick credit card. And, according to slides I've seen, compared to the 69GB SSD, the new modules give you a better per GB cost, more dense physical packaging and 50 percent less energy and heat per drive, with comparable performance.

As I've noted, filemon (filemon -O hot -A -x "sleep 20" -r fmon -o fmon.out) can help us identify which filesystems and physical and logical volumes should be moved to SSD drives and determine the proper physical locations for these relocated files.

If you're just starting to gather information about SSD technology, IBM offers some good introductory material. In particular check out the SSD vs. hard-disk drives comparison.

From IBM:

"Also known as Flash technology, solid-state drive technology eliminates the rotational delay of a spinning platter and of waiting for an arm to move to the correct position. Thus, data is available nearly immediately. Dramatically reducing crippling I/O bottlenecks, an SSD provides 33X to 125X more I/O Operations Per Second (IOPS) than a HDD and works at speeds much closer to those of memory, bridging the HDD performance gap. SSDs are also more efficient than HDD. While SSD operates close to 100 percent capacity, HDD is often limited to 20-50 percent storage capacity in an effort to improve responsiveness."

There's also a comparison of internal storage solutions and storage area networks (SANs) as well as a rundown of older SSD disk options, including SAS-connected 69 GB drives and the new double-wide PCIe card which can house up to four 177GB SSD eMLC disk modules.

Again, from IBM:

"eMLC technology stands for "Enterprise Multi-Level Cell" Flash memory technology. IBM is the first server vendor to provide this new SSD technology option which blends enterprise class performance and reliability characteristics with the more cost effective characteristics of MLC Flash storage."

Here's one more quote from IBM that I agree with wholeheartedly: "Remember, it’s not a question of if
solid-state drives will be part of your computer center, but rather, when."

Finally, revisit this post, which references Nigel Griffiths' SSD demonstration video.