i can

April 23, 2014

Health Indicators in the Performance Data Investigator

The Performance Data Investigator has a wealth of capabilities, some of which I have written about in the past. This week, I want to review the Health Indicators.

Health Indicators are a set of graphs (perspectives) found within the Investigate Data task in the Navigator for i Web console. The health indicators are based upon Collection Services data and are included with the base operating system (you do not need to purchase or install any additional licensed programs to use the Health Indicators). Health Indicators give you an overall view of the performance of your partition and can be used to easily determine whether you should investigate the performance of your partition further.


As you can see from the screen capture above, there are various different types of Health Indicators. The System Resources Health Indicators provide you an overview of high-level performance metrics for the partition (CPU, memory, disk, etc.). The other types of health indicators give you a bit more detail. Below is an example of the System Resoruces Health Indicators:


It’s important that you understand how to interpret the graph that’s displayed. For each metric that’s displayed, the colors (green, yellow, and red) represent the percentage of intervals within the collection that had values that were within that threshold level.

It’s easiest to explain by referring to the example above. If we look at the CPU metric, we will see that around 70% of the intervals, CPU metrics were healthy (green). About 10% of the intervals, we have values that were in the warning level (yellow), and 20% of the intervals, we had values that were above the action level (red). This graph does not tell us when the warning and action thresholds were hit – only that they occurred in some percentage of the intevals within the collection.

We can drill into the CPU health indicators to see more detail:


From this chart we can now see that we have concerns about 5% of the time regarding the partition CPU Utilization, but the major issue is Jobs CPU Queuing, which was happening around 80% of the time during the collection. To better understand what’s happening, we can now drill into the CPU Utilization and Waits overview to look at the Collection Services data over time, to see when these situations were occurring:


In the resulting chart, you can now see when the CPU Queuing time occurred and when the partition CPU utilization exceeded the Action threshold (circled in red).


These threshold values have defaults supplied by IBM, but you can modify them by taking the Define Health Indicators from the action drop-down:


The work management System Status GUI provides links to the Health Indicators. The General tab will provide a link to the System Resources Health Indicators, as shown below.


If you select other tabs from System Status, you will find links to other types of health indicators: the Processors tab includes a link to the CPU Health Indicators, the Memory tab includes a link to the Memory Health Indicators, and the Disk Space tab includes a link to the Disk Health Indicators. These buttons allow you to go from the work management system status information to view the Health Indicators based upon the current Collection Services data.

As previously mentioned, the Health Indicators are based upon Collection Services data; you can view the Health Indicators for the most recent collection (near real time information – i.e., how is the system performing now), as well as in the past for any collections you still have available.

In summary, you can use the Health Indicators to get an overview of the performance of your IBM i partition; if you see areas of concern, you can then use the Performance Data Investigator to drill into the performance data details to determine what issues may be occurring on your partition.



April 08, 2014

Istanbul International Tulip Festival

This past week has taken me to Turkey where I am a speaker at the IBM Systems Technical University held in in Istanbul. Since this is my first time traveling in this part of the world, I arrived a few days early to do some sightseeing. The first day was spent discovering the historic sites in Old Town. I stopped by Topkapi Palace, the Blue Mosqu, and the Hagia Sophia. I also visited the Museum of Archeology to check out their sarcophagi collection. On the way back to the hotel, there was an unexpected splash of color seen through the gates of a nearby park. I had noticed a lot of flower plantings while in the taxi from the airport, but the park was incredible! Elaborate plantings of tulips were everywhere and the park, which I learned was Gülhane Park, was filled with people enjoying the beauty. Almost everyone was in a happy mood with smiles on their faces.

I was curious about all the tulips, so I did a little research and found out that Istanbul has the International Tulip Festival each year. It’s a relatively new event that started in 2006. I also learned that tulips were originated in Turkey, not Holland as many may believe.

Below you’ll find a few pictures…






March 26, 2014

IBM i Storage Allocation Perspectives

Quite some time ago, I wrote a blog on Understanding Disk Space Usage. Near the end of that article, I mentioned that Collection Services collects storage usage data in the QAPMJOBMI file.

Late last year, there were additional enhancements to Navigator for i, which included updates to the Performance Data Investigator (PDI). With the Fall 2013 PDI enhancements, you can now visualize these Collection Services storage usage metrics. You will need to install the required PTFs to enable this support, as documented in the “Fall 2013” section of the Performance on the Web information.

You will now find some new charts that will show the storage usage on your system over time.

In PDI, open Collection Services then expand the Disk folder. Here you will find a new folder named Storage Allocation. In this folder are two new charts to display storage allocation information.


Storage Allocation/Deallocation Overview
This graph displays an overview of the storage allocated and deallocated for the partition overall; you can drill into storage allocation/deallocation by job from this overview.

Here’s an example of the graph:


Storage Allocation/Deallocation by Thread or Task

This graph displays the storage allocated or dealloacted by job or thread. You can then view the details for one job or thread over time.

Here is an example of this graph:


From here, we could even go look at the details of one of the jobs, where we can see how much storage the specific job allocated and deallocated over time.

Here’s an example:


If you think you have a storage leak in your system, the Collection Services data, along with the new support in PDI to visualize this information, is another useful diagnostic tool.

The wonderful thing about looking at Collection Services data is that you can see what has been happening over time. With interfaces such as Work with System Activity (WRKSYSACT), you see what the values are right now. Collection Services can help you understand what was happening in the past to help you understand how you got to where you are now.



March 18, 2014

IBM Knowledge Center

A few weeks ago, Steve Will wrote a couple blogs: IBM i Knowledge Center and IBM i 7.1 Knowledge Center, Redbooks, and developerWorks.

I’ve spent a little time exploring the Knowledge Center and this blog shares some highlights with you.

As Steve wrote, the Knowledge Center provides a single site for IBM documentation rather than the multiple information centers you may be using today. From an IBM i perspective, you can find the IBM i 6.1 and IBM i 7.1 information, as well as the Power information that was previously in the Hardware Information Center.


Since the Knowledge Center contains information from many products, you will most likely want to configure collections so you have a way to easily find (and search) the information that you are interested in.

In my example, I have set up a collection that includes IBM i 6.1, IBM i 7.1 and POWER7 hardware information. When you set up your own collections, you will need to sign in with your IBM ID. (Hint: the Sign In link is on the right-hand side of the upper black bar). At first I had some trouble with the IBM i information showing up as “Welcome” when I viewed my collection. The trick to getting it to show up as “IBM i 6.1” or “IBM i 7.1” is to navigate to the IBM i release you are interested in via the table of contents, right-click, and select “Save this product to an existing collection”, as the screen capture below shows.


When I go to My Collections, I see the one collection I have created named “My IBM Documentation”.


It’s great that you have one place to go to access IBM documentation, but the catch-22 is search. The Knowledge Center offers various ways to limit your searches and an article, Narrowing your searches in IBM Knowledge Center, discusses how you can do this. The IBM Technical Content blog has additional articles on the Knowledge Center capabilities that provide a useful overview of many features.

Bookmarks are another area where things have changed – they actually work! In the old information centers, when you navigated though the content, the URL would not change and bookmarks resulted in the main Information Center page, which is useless. You’d have to open the page in a new tab so it would have the complete URL and then you could bookmark the page. In the Knowledge Center, as you navigate through content, you’ll see the URL updated to reflect the page you are on and you can directly bookmark the page.

You can also share helpful content easily via the “Share” options. Perhaps I’ll be tweeting more as I’m reading things in the Knowledge Center since it will be so easy to do!


There are too many features to write about that are new or different from the prior Information Centers. Take some time to experiment with it. You might like it!

Finally, if you go to the IBM i 6.1 or 7.1 Information Centers, you will find a reminder to try out the Knowledge Center.




March 11, 2014

IBM i Performance Analysis

If you are a regular reader of this blog, you no doubt have come to realize how very fortunate we are on IBM i in the realm of performance tools and performance management. From the seamless instrumentation of metrics, to the various performance data collectors (all included in the operating system), to the powerful graphical tools that are available to help analyze the data. Couple that with the benefit of the patented wait accounting technology, and it's clear that we have an impressive end-to-end suite of technologies at our disposal to monitor, manage and solve performance problems.

Now on to the intangible – the methodology. Even with really great tooling, analyzing the performance of a partition can be a complex task. The amount of data can be overwhelming. It can be hard to know where to start, what tool(s) to use, and whether the data is indicative of a problem or not. Does this sound familiar?

Let’s imagine the following scenario. Every day this week, the users have started to complain about their response times becoming unreasonable around 3:00 p.m., and the slowdown usually last for an hour. A typical methodology to use in the scenario would be to use leverage the real-time tools (Active Jobs, Disk Status, System Status, etc.) to gain initial clues on what system resources are being consumed (and a fleeting look at by who) during the particular troublesome timeframe. The next step would be to use the capabilities of Collection Services to determine in further detail who is using resources and what resources are being consumed. In addition to the who/what is being used, Collection Services is also used to find bottlenecks such as what resources jobs are waiting for (e.g., CPU, Disk I/O), and types of contention time if present. Sometimes, the problem can be solved with just this information. Perhaps during this timeframe we see a lot of CPU queuing time occurring. So next, we look at who is using CPU, and because we kept a baseline (you are keeping a baseline, right?), we look for changes. What we notice is that a new set of reports for management are now being run at 3 p.m. every day, instead of during off-shift hours, pushing us over the “knee of the curve” for CPU utilization. In this case, it may be as simple as rescheduling these reports. However, unfortunately, in many cases it is not so simple and we must continue the investigation. The next step then would be to focus our attention on Job Watcher data. If not already being collected, we begin to collect it. Job Watcher will give us an even more granular view of where time is being spent both at a system level and at a job level. It will give us additional clues on wait objects, call stacks and SQL being run. If the problem is not solved using Job Watcher, but we can establish it is I/O related, we may next turn to Disk Watcher. If we need to get down into the application and program level in order to completely solve the issue, we will use Performance Explorer.

Now, back to my original question “Does this sound familiar?”…..

Because we know that many of you nodded your head “yes”, the Lab Services IBM i Performance team was inspired to build a three-day workshop “IBM i Performance Analysis Workshop” to help teach you some of the techniques we use to analyze data and solve performance problems. This workshop is not focused around how to use the tools, but rather the methodology used. We will cover the strengths of the collector tools and when to use them, as well as core methodology and analysis techniques for CPU, Memory, I/O and various waits. Our goal is to aid you in building a foundation of performance methodology you can apply in your environment. If you are interested in more details, including enrolling in a public workshop, please see this link:

We also do custom on-site workshops as a Lab Services offering. If you are interested, please contact Mike Gordon at [email protected].

I’d like to thank Stacy Benfield for writing this blog. Stacy is a member of the Lab Services Performance team and was actively involved in the creation of this workshop. Prior to joining the Lab Services performance team, Stacy was part of the IBM i development team, working on performance tools, where she was the team leader of the Performance Data Investigator.