TCP/IP Redundancy with Virtual Ethernet in IBM i

This week's blog was written by Colin DeVilbiss. Colin develops Communications and Virtual I/O support for IBM i in Rochester, MN. Thanks, Colin!

IBM i 7.1 Technology Refresh 3 added support for Ethernet Link Aggregation, as previously featured in the i Can blog "Ethernet Link Aggregation in IBM i" and the tip IBM i Aggregate Ethernet Lines on IBM developerWorks. An aggregate line description combines multiple physical Ethernet ports. The line description can then be used for TCP/IP configuration just like any other line description, but the incoming and outgoing traffic will be spread among the available ports. As a side benefit, the line description will remain active as long as any of the aggregated ports are available.

Link aggregation is a great redundancy solution for IBM i partitions that have multiple physical Ethernet ports available, all of which can be connected to a single Ethernet switch that has support for link aggregation. However, as some have commented on those articles, link aggregation does not work with virtual Ethernet ports that are connected to the internal Ethernet switch of the Power Hypervisor. This means that it is not useful to partitions that connect to the network via a Shared Ethernet Adapter (SEA) in a Virtual I/O Server (VIOS) partition or via an Ethernet layer-2 bridge provided by an IBM i partition (which was also added in Technology Refresh 3).

 

SEA Failover

One solution is to use a pair of VIOS partitions, each with an SEA bridged to the physical network, and to have the Shared Ethernet Adapters configured for failover, which provides redundancy without any additional configuration in the bridged partitions.

 

Virtual IP Addressing

Bridged IBM i partitions can get Ethernet redundancy from the existing Virtual IP Address (VIPA) support provided by IBM i. Specifically, I will describe solutions based on VIPA Proxy ARP.

Unlike other IP interfaces, a VIPA is not bound to a line description. Instead, the user chooses a list of existing IP interfaces that will serve as proxies to connect the VIPA to the network; these are called its preferred interfaces. They are kept in priority order, and the VIPA will be serviced by the first active interface in the list.

 

Physical Interface With Virtual Backup

Given this, we can build a setup that will default to using a physical Ethernet port but will use a virtual Ethernet port instead if the physical port becomes unavailable. For this example, we assume that the IP addresses below are all available and appropriate to the system's network, the physical Ethernet port is managed by line description PHYSETH, the virtual Ethernet port is managed by line description VIRTETH, and that VIRTETH is bridged to the physical network by a VIOS SEA or IBM i layer-2 bridging. We create an IP interface on each port, then create a VIPA, preferring the physical interface over the virtual Ethernet interface.

 

ADDTCPIFC INTNETADR('192.168.1.2') LIND(PHYSETH) SUBNETMASK('255.255.255.0')

ADDTCPIFC INTNETADR('192.168.1.3') LIND(VIRTETH) SUBNETMASK('255.255.255.0')

ADDTCPIFC INTNETADR('192.168.1.4') LIND(*VIRTUALIP) SUBNETMASK('255.255.255.255')

    PREFIFC('192.168.1.2' '192.168.1.3')

 

Once all three interfaces are started, the VIPA should be accessible from the network via PHYSETH; if PHYSETH fails, then VIRTETH will take over.

When each partition has a physical interface available, this can be a great solution. Each partition has control over its own physical Ethernet resource, and a single bridging support partition can provide a backup for all of the other partitions simultaneously.

For direct use, this requires that the first line description in the preferred interface list will fail if it can't access the network (for example, if the cable is pulled or the Ethernet adapter or switch fails for some reason). This is true for IBM i physical Ethernet ports (and for aggregate line descriptions) but not true for bridged virtual Ethernet ports (which are not connected directly to the network). Thus, this solution requires changes if we want to support failover between two virtual Ethernet ports.

 

Virtual Interface With Virtual Backup

This can be done by writing a program that does some level of direct health check for each preferred interface and rearranges the preferred interfaces so that the first is always working. Attached below is an example CL script that monitors each preferred interface to see that it can ping a specific remote IP address, and changes the priority order if the first interface becomes unavailable while the second is available.   Download 121912 monitor

Given this, we can fail over between two virtual Ethernet ports. This enables more complicated configurations that provide Ethernet redundancy for the clients without setting up any failover between bridging partitions.

For example, consider a system with several IBM i partitions, two of which have physical Ethernet resources. In each of those partitions, we configure a layer-2 bridge: the first bridge through one Power Hypervisor virtual LAN (VLAN), and the second through a different VLAN. Then, in each of the other IBM i partitions, we configure two virtual Ethernet interfaces and VIPA as above, with a virtual-to-virtual failover script. In half of those partitions, we choose the first bridge as primary; in the other half, we make the second bridge primary.

In this system, all of the physical Ethernet resources are being used, and the physical network is unaware of anything in the configuration. If one bridge stops working for any reason, the half of the partitions that were using it will start using the other bridge. This can provide acceptable service on systems with several IBM i partitions but not a lot of available physical Ethernet resources, at the cost of some additional configuration work.

In short, IBM i and Power Systems network and virtualization technologies give users the tools they need to meet their performance and redundancy targets. We are always looking for ways to improve our networking options as well.

 

 

Ethernet Link Aggregation in IBM i

This week’s blog was written by Colin DeVilbiss, an IBM software engineer. He works mostly on communications device drivers for IBM i in the development lab in Rochester, Minn. Thanks, Colin.

In 7.1 Technology Refresh 3, IBM i added the capability to aggregate multiple Ethernet ports in one Ethernet line description. An aggregated line description offers two significant benefits over a single-resource line description: improved reliability and parallel throughput.

An aggregate line description stays available as long as at least one of its aggregated Ethernet ports is working; for example, if four Ethernet ports are aggregated in one line description, the line description will remain active even if three of those Ethernet ports lose their link, and all of the workloads will continue running.

In addition, workloads with multiple parallel TCP connections or with multiple remote clients will be able to take advantage of having multiple physical connections to the same IP address, increasing throughput with each available resource. For example, if four Ethernet ports are aggregated, each running at one gigabit per second, the aggregate could support up to four gigabits of throughput across multiple connections.

 

New Capabilities

Link aggregation gives IBM i administrators a new tool to improve their network configurations, taking advantage of underutilized resources and adapting to meet new demands in the business.

For example, if a partition has a two-port Ethernet adapter, but IBM i is only using one port, it's straightforward to aggregate the two ports together and have additional protection against a failure in a cable, switch port, or port on the adapter, while potentially doubling throughput on some workloads, potentially with no additional hardware cost.

Likewise, if an IP interface is getting close to saturating its capacity, link aggregation lets the administrator scale the underlying line description's capacity incrementally, without changing technology for the individual links. That is, if one gigabit per second isn't enough, link aggregation can provide two gigabits per second of potential throughput without needing to jump to 10-gigabit-per-second infrastructure.

 

Creating an Aggregate Line Description

The main requirements for creating an aggregate line description are:

• An IBM i partition at 7.1 with Technology Refresh 3 and its PTF group loaded
• Two or more gigabit-capable physical Ethernet resources in the partition
• A switch that supports link aggregation (sometimes under the names “trunking” or “teaming”)

At the switch, the corresponding ports must be configured into a static aggregate; switches vary, and each manufacturer has its own configuration commands and interfaces, so check there.

Assuming that the desired Ethernet resources are CMN02 and CMN03, creating the aggregate line description is as simple as:

CRTLINETH LIND(ETHAGG) RSRCNAME(*AGG) AGGPCY(*ETHCHL *SRCDESTP) AGGRSCL(CMN02 CMN03)

That is, create a new Ethernet line description

• named ETHAGG
• using link aggregation instead of a single hardware resource
• using static assignment of resources (the only supported policy in TR3)
• assigning outgoing frames to ports based on the source and destination IP addresses and TCP ports (which is likely to provide the best “spreading” of traffic across the ports)
• aggregating CMN02 and CMN03

Varying the line description on will bring all of the ports active, and DSPLIND can be used to see the status of each port. TCP/IP can use the line description just like any other Ethernet line description.

For more details, see the Ethernet on System i topic in the IBM Information Center.

In short, Ethernet Link Aggregation lets 7.1 TR3 users make better use of their Ethernet resources, potentially improving the reliability and performance of their network workloads without purchasing any additional hardware.