Health Indicators in the Performance Data Investigator

The Performance Data Investigator has a wealth of capabilities, some of which I have written about in the past. This week, I want to review the Health Indicators.

Health Indicators are a set of graphs (perspectives) found within the Investigate Data task in the Navigator for i Web console. The health indicators are based upon Collection Services data and are included with the base operating system (you do not need to purchase or install any additional licensed programs to use the Health Indicators). Health Indicators give you an overall view of the performance of your partition and can be used to easily determine whether you should investigate the performance of your partition further.

As you can see from the screen capture above, there are various different types of Health Indicators. The System Resources Health Indicators provide you an overview of high-level performance metrics for the partition (CPU, memory, disk, etc.). The other types of health indicators give you a bit more detail. Below is an example of the System Resoruces Health Indicators:

It’s important that you understand how to interpret the graph that’s displayed. For each metric that’s displayed, the colors (green, yellow, and red) represent the percentage of intervals within the collection that had values that were within that threshold level.

It’s easiest to explain by referring to the example above. If we look at the CPU metric, we will see that around 70% of the intervals, CPU metrics were healthy (green). About 10% of the intervals, we have values that were in the warning level (yellow), and 20% of the intervals, we had values that were above the action level (red). This graph does not tell us when the warning and action thresholds were hit – only that they occurred in some percentage of the intevals within the collection.

We can drill into the CPU health indicators to see more detail:

From this chart we can now see that we have concerns about 5% of the time regarding the partition CPU Utilization, but the major issue is Jobs CPU Queuing, which was happening around 80% of the time during the collection. To better understand what’s happening, we can now drill into the CPU Utilization and Waits overview to look at the Collection Services data over time, to see when these situations were occurring:

In the resulting chart, you can now see when the CPU Queuing time occurred and when the partition CPU utilization exceeded the Action threshold (circled in red).

These threshold values have defaults supplied by IBM, but you can modify them by taking the Define Health Indicators from the action drop-down:

The work management System Status GUI provides links to the Health Indicators. The General tab will provide a link to the System Resources Health Indicators, as shown below.

If you select other tabs from System Status, you will find links to other types of health indicators: the Processors tab includes a link to the CPU Health Indicators, the Memory tab includes a link to the Memory Health Indicators, and the Disk Space tab includes a link to the Disk Health Indicators. These buttons allow you to go from the work management system status information to view the Health Indicators based upon the current Collection Services data.

As previously mentioned, the Health Indicators are based upon Collection Services data; you can view the Health Indicators for the most recent collection (near real time information – i.e., how is the system performing now), as well as in the past for any collections you still have available.

In summary, you can use the Health Indicators to get an overview of the performance of your IBM i partition; if you see areas of concern, you can then use the Performance Data Investigator to drill into the performance data details to determine what issues may be occurring on your partition.

 

 

IBM i Storage Allocation Perspectives

Quite some time ago, I wrote a blog on Understanding Disk Space Usage. Near the end of that article, I mentioned that Collection Services collects storage usage data in the QAPMJOBMI file.

Late last year, there were additional enhancements to Navigator for i, which included updates to the Performance Data Investigator (PDI). With the Fall 2013 PDI enhancements, you can now visualize these Collection Services storage usage metrics. You will need to install the required PTFs to enable this support, as documented in the “Fall 2013” section of the Performance on the Web information.

You will now find some new charts that will show the storage usage on your system over time.

In PDI, open Collection Services then expand the Disk folder. Here you will find a new folder named Storage Allocation. In this folder are two new charts to display storage allocation information.

Storage Allocation/Deallocation Overview
This graph displays an overview of the storage allocated and deallocated for the partition overall; you can drill into storage allocation/deallocation by job from this overview.

Here’s an example of the graph:

Storage Allocation/Deallocation by Thread or Task

This graph displays the storage allocated or dealloacted by job or thread. You can then view the details for one job or thread over time.

Here is an example of this graph:

From here, we could even go look at the details of one of the jobs, where we can see how much storage the specific job allocated and deallocated over time.

Here’s an example:

If you think you have a storage leak in your system, the Collection Services data, along with the new support in PDI to visualize this information, is another useful diagnostic tool.

The wonderful thing about looking at Collection Services data is that you can see what has been happening over time. With interfaces such as Work with System Activity (WRKSYSACT), you see what the values are right now. Collection Services can help you understand what was happening in the past to help you understand how you got to where you are now.

 

 

IBM i Performance Analysis

If you are a regular reader of this blog, you no doubt have come to realize how very fortunate we are on IBM i in the realm of performance tools and performance management. From the seamless instrumentation of metrics, to the various performance data collectors (all included in the operating system), to the powerful graphical tools that are available to help analyze the data. Couple that with the benefit of the patented wait accounting technology, and it's clear that we have an impressive end-to-end suite of technologies at our disposal to monitor, manage and solve performance problems.

Now on to the intangible – the methodology. Even with really great tooling, analyzing the performance of a partition can be a complex task. The amount of data can be overwhelming. It can be hard to know where to start, what tool(s) to use, and whether the data is indicative of a problem or not. Does this sound familiar?

Let’s imagine the following scenario. Every day this week, the users have started to complain about their response times becoming unreasonable around 3:00 p.m., and the slowdown usually last for an hour. A typical methodology to use in the scenario would be to use leverage the real-time tools (Active Jobs, Disk Status, System Status, etc.) to gain initial clues on what system resources are being consumed (and a fleeting look at by who) during the particular troublesome timeframe. The next step would be to use the capabilities of Collection Services to determine in further detail who is using resources and what resources are being consumed. In addition to the who/what is being used, Collection Services is also used to find bottlenecks such as what resources jobs are waiting for (e.g., CPU, Disk I/O), and types of contention time if present. Sometimes, the problem can be solved with just this information. Perhaps during this timeframe we see a lot of CPU queuing time occurring. So next, we look at who is using CPU, and because we kept a baseline (you are keeping a baseline, right?), we look for changes. What we notice is that a new set of reports for management are now being run at 3 p.m. every day, instead of during off-shift hours, pushing us over the “knee of the curve” for CPU utilization. In this case, it may be as simple as rescheduling these reports. However, unfortunately, in many cases it is not so simple and we must continue the investigation. The next step then would be to focus our attention on Job Watcher data. If not already being collected, we begin to collect it. Job Watcher will give us an even more granular view of where time is being spent both at a system level and at a job level. It will give us additional clues on wait objects, call stacks and SQL being run. If the problem is not solved using Job Watcher, but we can establish it is I/O related, we may next turn to Disk Watcher. If we need to get down into the application and program level in order to completely solve the issue, we will use Performance Explorer.

Now, back to my original question “Does this sound familiar?”…..

Because we know that many of you nodded your head “yes”, the Lab Services IBM i Performance team was inspired to build a three-day workshop “IBM i Performance Analysis Workshop” to help teach you some of the techniques we use to analyze data and solve performance problems. This workshop is not focused around how to use the tools, but rather the methodology used. We will cover the strengths of the collector tools and when to use them, as well as core methodology and analysis techniques for CPU, Memory, I/O and various waits. Our goal is to aid you in building a foundation of performance methodology you can apply in your environment. If you are interested in more details, including enrolling in a public workshop, please see this link:

http://www-03.ibm.com/systems/power/software/i/support/workshops/performance-analysis.html

We also do custom on-site workshops as a Lab Services offering. If you are interested, please contact Mike Gordon at [email protected].

I’d like to thank Stacy Benfield for writing this blog. Stacy is a member of the Lab Services Performance team and was actively involved in the creation of this workshop. Prior to joining the Lab Services performance team, Stacy was part of the IBM i development team, working on performance tools, where she was the team leader of the Performance Data Investigator.

 

 

Application Runtime Expert – Damaged Object Detection

A few weeks ago, I wrote about the Application Runtime Expert – Network Health Checker. IBM recently made another utility available via the Application Runtime Expert that helps you identify damaged objects.

Objects can get damaged for a variety of reasons and you may not know you have damaged objects until you actually access the object (or the part of the object) that is damaged. One of the more common reasons for damage is a disk failure of some sort – a disk controller failure for example; in the case of disk failures, some of the data may not have been correctly written to disk resulting in damage.

You need to identify damaged objects so you don’t lose data; the typical way to do this has been to perform a full system save. However, if you’re not sure whether you have damaged objects, you have to save everything to cause the system to read all parts of all objects. However, there is a catch-22: if the save encounters a damaged object, the save itself will fail, thus making it difficult to identify all the damaged objects. The damage detection tool can make this process easier.

The use of this utility is very similar to that of the Network Health Checker that I wrote about previously. There is a new parameter on the areVerify script. To run the damaged object detection tool, you go into QShell and use the following incantation (note that you need to have *ALLOBJ special authority):

/QIBM/ProdData/OS/OSGi/templates/bin/areVerify.sh –storage diskUnits=*ALL

You can also request a check for damage only on specific disk units. For example:

/QIBM/ProdData/OS/OSGi/templates/bin/areVerify.sh -storage diskUnits=1,2,4

While the damage detection utility is running, you will find several jobs get submitted to the QCTL job queue. In my example, they run under my user profile since I initiated the utility. You will see something like the following for the jobs that are submitted:

The jobs run at a priority such that normal work on the system should not be affected. However, because the damage detection tool is reading everything on disk, you will notice increased I/O activity.

A report is generated with the results of the check – it is stored in /tmp and is named areDodReport.txt. If damage is detected, the report will include information about the damaged object(s). Of course, my test system was uninteresting and I had no damaged objects.

Most of you are probably more comfortable looking at this via WRKLNK, as the screen capture below shows.

As I discussed earlier in this blog, this utility identifies one type of damage – it’s called a data check. At this time, it will not identify all types of damage for you since there are many reasons damage can occur. However, if you’ve had a disk failure, this utility is a great approach to check for damage.

The article Automate Damaged Object Detection on Your System, written by Tim Rowe, provides more details that you may find useful.

This enhancement is available on both 7.1 and 6.1. It was delivered via PTFs. The information below lists the PTFs that you will need. Order the PTF in bold and include the requisites so you will get all the fixes you need in one order.

• 7.1 - SI50374 (5770SS1)
• Make sure the following requisite are also applied:
• MF56898
• MF56876
• SI45469

For 7.1, the HTTP Server group PTF (SF99368) includes these PTFs; if you install at least level 21 of this group, you will have the above PTFs.

• 6.1 - SI45499 (5761SS1)
• Make sure the following requisites are also applied:
• SI51025 (5761SS1)
• SI30796 (5761SS1)
• R610:
• MF57435 (5761999)
• MF57436 (5761999)
• R611:
• MF57425 (5761999)
• MF57426 (5761999)

Schedule Verifications With Application Runtime Expert

IBM has made another set of announcements and among them is IBM i 7.1 Technology Refresh 7. As part of the TR7 enhancements, the ability to schedule verifications has been added to the Application Runtime Expert.

From the IBM i TR7 RFA:

“Scheduling flexibility enhancements have been made to IBM Application Runtime Expert.”

The Application Runtime Expert product (5733-ARE) has been enhanced to include the ability to schedule verifications based on a timeframe that fits your needs. From the Application Runtime Expert GUI interface, it is possible to schedule a verification to run at a specific time against a list of chosen systems. This provides additional flexibility and usability to help extend the usefulness of the product. This new support will be delivered in the latest HTTP PTF Group and will require that the 5733-ARE product is already installed. These new enhancements are available for both IBM i 6.1 and IBM i 7.1.

The Application Runtime Expert is easy to use and the capability to schedule verifications makes it even easier; you no longer have to remember to manually run verifications. You can set up groups of systems with the attributes you want to have validated and specify the scheduling frequency; once it is configured, the verifications will be done automatically for you. All you have to do is review the resulting reports from the verification.

When you go to the console to verify the systems you have selected, there is now a scheduling icon.  Click on the Schedule link to open the scheduling window where you can select when you want the verification to automatically run. You can schedule verifications on a daily, weekly or monthly schedule.

The scheduling support is part of the Application Runtime Expert HTTP server, so the IBMARE server must be running in order for the scheduled verification to occur.

As a reminder, the HTTP group PTFs are:

• 7.1 - SF99368

• 6.1 - SF99115

The Application Runtime Expert for i website contains a lot of useful information.

I’ve also written about the application runtime expert before in the following blogs:

Application Runtime Expert – Network Health Checker

The Story of the Application Runtime Expert

Finally, be sure to read the following blogs to learn about other content in this latest announcement

• You and i from Steve Will
• iSight from Alison Butterill
• Modern-i-zation from Tim Rowe
• DB2 for i from Mike Cain
• RPG Café Blog where Barbara Morris contributes about RPG topics.

 

Application Runtime Expert - Network Health Checker

I’ve written a bit about the Application Runtime Expert (which you will often see referenced as ARE) previously.

As part of the Application Runtime Expert support, IBM has shipped a few basic ARE templates that can be used without requiring you to have the Application Runtime Expert product (5733ARE) on your system. A template defines the environmental information – such as configuration setting or attributes – that are expected. There are currently have four templates available as part of the operating system. All of these can be run on an IBM i partition from Qshell. You need the HTTP group PTF (SF99115 on IBM i 6.1, or SF99368 on IBM i 7.1) installed.

One specific utility (template) that I want to tell you about is the network health checker. This network configuration plug-in for ARE verifies a variety of TCP/IP configuration settings and network characteristics. It checks for common configuration issues, much like reviewing the information available via the CFGTCP command. This template also performs basic network operations, such as looking up localhost and checking if all configured DNS servers can be reached. You can run this template to verify that the system's network configuration is working in a fast, reliable, and repeatable manner.

Here’s a practical example ….

I use Navigator for i quite a bit - in particular, the performance tasks, but for other things as well. I was trying to use Navigator on an IBM i partition that I’d never worked with before and it was SSSSLLLOOOOWWW! Of course, the first thing one tends to look at is the CPU and memory configuration for the partition – but these were fine – uncapped to 2 cores with 10GB of memory – plenty to run Navigator. So the next step was to see if I had any networking issues.

I decided to give the ARE network health checker a try.

You need to use QShell to run these templates. There are various ways you can do this, but perhaps the simplest is to simply start the QShell environment with STRQSH and work from there.

To run the network health checker, you use the following incantation:

/QIBM/ProdData/OS/OSGi/templates/bin/areVerify.sh –network

The network health checker immediately identified the problem – retrieving the local host name took a long time – 28 seconds! The text below was the information returned from the network health checker that clearly identified the problem (I have highlighted the problem in red).

 

Running plugin Network Verifier 

  Retrieving local host name

     Start time is Tue Aug 27 16:39:06 CDT 2013 

    Local host name is etcsdi1s.rchland.ibm.com 

     End time is Tue Aug 27 16:39:34 CDT 2013

  Warning: Retrieving local host name took longer than 10 seconds.  Elapsed time was 28 seconds     

  Details: Taking this long to resolve localhost can lead to slow network performance. 

The rest of the output from the network health checker gave me information about the network configuration – so it was very easy to compare the output from the system with very poor performance to that of a system that was working well. Sure enough – the DNS server settings were incorrect for the partition with the performance issue. I corrected the DNS server configuration and my problem was resolved.

Note that the Network Health Checker is also available via the ARE graphical user interface; however, as I’ve just written, you do not need the ARE product to use this feature.

There are a few other health checkers available with the areVerify.sh script. Invoking this script without any parameters will return the following:

 

/QIBM/ProdData/OS/OSGi/templates/bin/areVerify.sh

Usage: areVerify.sh <parameter>

Valid parameter:

    -network               : Verify network configuration 

                             and status

    -iasserver server_name : Verify the 'server_name' 

                             IAS server

    -precheck              : Verify software products 

                             that are necessary to use 

                             the IBM Application Runtime 

                             Expert for i

    -hostservers           : Verify the host servers are 

                             active

 

Finally, here are a couple useful websites with additional information on the Application Runtime Expert:

Application Runtime Expert

Revolutionize your application and product support

 

 

 

Viewing Job Level SQL Metrics with the Performance Data Investigator

Previously I wrote about Job Level SQL Metrics in Collection Services. In this week’s blog, I will review the support added to the Performance Data Investigator (PDI) to allow you to view and analyze these job-level SQL metrics.

First, you must have the latest Navigator for i PTFs installed as reviewed in the blog Navigator for i Enhancements. You will also want to verify that you have the PTFs installed and activated that were noted in the Job Level SQL Metrics in Collection Services blog.

The GUI for the new SQL metrics is contained in a content package named Database. This content package will only show up under the list of perspectives in the Performance Data Investigator if you have the Performance Tools licensed program product (5770-PT1) installed; you just need the manager option. Put another way, the enablement for the database graphical interface in the Performance Data Investigator is shipped with PT1.

Once you have the latest Navigator PTFs installed (along with the performance tools and database group PTFs that I reviewed previously), you will discover you now have some new perspectives available in the Performance Data Investigator. In particular, you will notice a new Database folder that includes Database I/O perspectives and SQL Performance Data perspectives. The following screen capture shows this:

You need to drill down into the Database I/O and subsequent folders to find the graphs and charts of data you can now view on Physical Database I/O metrics – as you can see from the following screen capture, you have several views of these database metrics:

If you have not followed the activation instructions required for the Collection Services PTF I previously wrote about, you will get an error if you try to view one of these physical database I/O charts. This is because the underlying data is not collected. Refer to SI47594 PTF Special Instructions for details.

An example of the Physical Database I/O Overview chart is below. You can see the SQL-related physical I/O metrics are summarized by time for the selected collection.

There are several different ways you can view the physical database I/O – by job, thread, user profile, current user profile, subsystem, or server type – to give you various ways to view and analyze the metrics.

You will note that the Performance Data Investigator now also supports viewing data from the SQL plan cache and the SQL performance monitor. I don’t have time in this blog to go into details on using PDI to look at this data, but for those of you who are interested in database performance, it’s worth getting current on your PTFs and checking out these new features.

The Performance Data Investigator development team has created documentation on developerWorks that summarizes the latest enhancements that have been made. There is an article on the new Database content package that you can look at for more information.

Finally, there are some recent changes that were not discussed in the prior blog regarding the SQL metrics. My previous blog stated that this was only supported on the 7.1 release, but the database metrics are now also available on the 6.1 release! Please read the Spring 2013 documentation (which is a subset of the Performance on the web documentation) on the IBM i developerWorks site to get the details regarding the support added to 6.1 and the required PTFs. If you read that Web page, you’ll discover that I have even more to blog about in coming weeks!

 

 

Job Level SQL Metrics in Collection Services

Late last year, IBM enabled new support to collect SQL metrics in Collection Services. I’ll be covering this new capability in two blog articles:

1. This blog will review the new metrics that are now collected and describe what you need to do to enable the collection of these additional metrics.
2. A subsequent blog will review the support added to the Performance Data Investigator to allow you to view and analyze these metrics.

Collection Services collects a wealth of information, but metrics related to SQL operations were limited.  The tools used for DB2 and SQL performance analysis are separate from the system performance tools; interfaces such as the DB2 Health Center, SQL Plan Cache and Index Advisor, among others, are what you need to use to understand database performance issues.

The most recent PTFs augment the tools already available for SQL performance analysis and there are now SQL metrics collected at the job level. This allows you to begin your performance analysis with Collection Services and determine if SQL operations are a contributing factor. You will probably still use the DB2-specific tools for digging into the details, but you now have insight into the SQL operations from Collection Services data.

This enhancement is only available on IBM i 7.1 (i.e., it is not available on 6.1) and requires that you have installed the latest DB2 Group PTF SF99701, as well as SI47594, which provides the Collection Services enablement. It’s also a good idea to ensure that you have the latest Performance Tools Group PTF, SF99145, installed as well.

Just installing the required PTFs is not sufficient. There are activation instructions you must take to enable the collection of these metrics documented in the SI47594 PTF cover letter. The instructions are rather lengthy but not difficult to follow. The Job Level SQL Metrics article on developerWorks also documents these activation instructions.

The reason for these special instructions has to do with the fact that the file, QAPMJOBMI, needs to be updated to a new level to support these additional metrics.  Specifically, an updated template file for QAPMJOBMI is shipped in QSYS with the additional fields. This template file will only be used when creating a new QAPMJOBMI file in libraries where it doesn't already exist. Existing collection libraries (for example, QPFRDATA) that contain a prior version of QAPMJOBMI will not be affected by the PTF.

Once you have the required PTFs installed and completed the necessary activation instructions, you will have SQL metrics in your job-level Collection Services data. The Performance Data Investigator can then be used to view and analyze these metrics. As I mentioned earlier, I’ll write more about the PDI support in a future blog. To be prepared to use this new capability, review the blog Navigator for i Enhancements and install the PTFs as documented in that blog.

The following are the new metrics now available for collection:

New Collection Services metrics for SQL:

The following SQL performance metrics are collected as part of the *JOBMI category.

1. SQL clock time (total time in SQ and below) per thread (microseconds)
   The amount of clock time (in microseconds) this thread has spent performing work done on behalf of an SQL operation.
2. SQL unscaled CPU per thread (microseconds)
   Thread unscaled SQL CPU time used. The amount of unscaled processor time (in microseconds) this thread has used performing work done on behalf of an SQL operation.
3. SQL scaled CPU per thread (microseconds)
   Thread scaled SQL CPU time used. The amount of scaled processor time (in microseconds) this thread has used performing work done on behalf of an SQL operation.
4. SQL synchronous database reads per thread
   Total number of physical synchronous database read operations done on behalf of an SQL operation.
5. SQL synchronous non-database reads per thread
   Total number of physical synchronous nondatabase read operations done on behalf of an SQL operation.
6. SQL synchronous database writes per thread
   Total number of physical synchronous database write operations done on behalf of an SQL operation.
7. SQL synchronous non-database writes per thread
   Total number of physical synchronous nondatabase write operations done on behalf of an SQL operation.
8. SQL asynchronous database reads per thread
   Total number of physical asynchronous database read operations done on behalf of an SQL operation.
9. SQL asynchronous non-database reads per thread
   Total number of physical asynchronous nondatabase read operations done on behalf of an SQL operation.
10. SQL asynchronous database writes per thread
    Total number of physical asynchronous database write operations done on behalf of an SQL operation.
11. SQL asynchronous non-database writes per thread
    Total number of physical asynchronous nondatabase write operations done on behalf of an SQL operation.
12. Number of high Level SQL Statement count per thread
    The number of high level SQL statements that executed during the Collection Services time interval. This count includes only initial invocation, independent SQL statements. It does not include dependent SQL statements invoked from within another SQL statement. This count also includes initial invocation, independent SQL statements which failed to execute successfully.

 

 

 

 

Performance Data Collectors in IBM i

There were significant changes to the IBM i performance tools in the 6.1 release. The ability to collect additional types of performance data, management of this data, and a graphical interface for viewing and analyzing this data were added. I often speak on performance topics, but I have not written much about them. Since there is much to cover, this will be the first in a series of articles. A good place to start is to review the performance data collectors available on i.

Beginning with the 6.1 release, there are four different types of performance data you can collect; two of them newly integrated into the operating system with that release. IBM i has the richest set of performance data compared to any other operating system; I believe it is a substantial differentiator, giving i users greater insight into how their machines are functioning.

 

Collection Services

Collection Services has been part of the operating system for many releases. Most administrators should be aware of it and using it regularly. Collection Services collects performance data from throughout the operating system on an interval basis and is designed to be running continuously. The overhead for data collection is not significant, but you do need to carefully manage the data that is collected. The major change with Collection Services in the 6.1 release is that it was changed to always be running by default.

Collection Services is used for day-to-day management of your system's performance; you use it for your baseline performance data collection to more quickly determine if changes have occurred that affect performance. It's the underlying data for Management Central System Monitors, it's the data that is sent to IBM if you use PM for Power Systems, and it's also the data that's used for trending and capacity planning purposes.

 

Performance Explorer

Performance Explorer, commonly referred to as PEX, has been part of the operating system since the V3R6 release. PEX has three modes of operation and can collect statistical performance data as well as trace data. Today, PEX is generally used for two reasons: 1) as a performance collector to aid with detailed application performance analysis, and 2) to collect detailed performance for diagnostics on the most difficult performance problems. PEX used to be used much more, but the development of the Job Watcher performance data collector has changed that.

 

Job Watcher

Job Watcher first appeared around the V5R1 time frame. Job Watcher was initially developed by the Rochester Support Center as a tool to aid performance diagnostics. Job Watcher and the graphical user interface used to analyze the data proved to be very powerful for performance diagnostics. The iDoctor offering was the first place where Job Watcher was made available to our clients. Since Job Watcher was so effective at performance diagnostics, it was built into the operating system with the 6.1 release.

Job watcher can collect much of the same information that Collection Services does, but it collects much more; call stacks, SQL statements, wait information (see i Can ...Tell You Why You Are Waiting for more information about waits), activation group information, Java information and more. In addition, while Job Watcher uses a sampling technology similar to that of Collection Services, Job Watcher has the option to sample as fast as possible, which can provide data similar to a trace.

Job Watcher is the most commonly used performance data for detailed diagnostics. The Rochester Support Center performance analysts, as well as the performance consultants in IBM Lab Services generally use Job Watcher. Nearly every performance problem can be resolved with Job Watcher data.

 

Disk Watcher

Disk Watcher is a specialized performance data collector that collects data specifically for disk operations. Its use is intended for detailed diagnostics of disk performance problems or for the use in optimizing disk I/O in applications.

All of these performance data collectors have a set of commands that you can use to define the data you want to collect, commands to start and end the data collection, and commands to manage the data. There are graphical user interfaces that I'll cover in the future as well.

A prior blog article on Find All my Performance Collections in One Place reviews the Collections task in the Navigator for i Web console. The Collections task can be used to see all of your performance collections, regardless of what type, in one consolidated interface.

As I noted at the beginning of this article, IBM i's performance data collectors are second to none; the combination of Collection Services for day-to-day performance management and Job Watcher for detailed diagnostics provide a powerful combination of tooling that can’t be beat.

 

 

 

Work With Traces

In the 5.4 release, IBM introduced the Work with Traces (WRKTRC) command. This is a handy little utility that allows you to review all the traces that may be active on a partition, as well as a centralized interface for managing traces.

Why is the WRKTRC a convenient tool? IBM i has very strong diagnostic capabilities and one of the reasons is due to the wide variety of diagnostic data that you can collect. However, different types of diagnostic data are often collected in different ways. In fact, there are several different trace facilities available on i:

• Job Trace
  Job trace (STRTRC) traces the program flow (among other things) within a job. I blogged about job trace some time ago in the article IBM i Job Trace. The Trace Job (TRCJOB) command still exists, but it now invokes STRTRC automatically. A little secret about Job Trace is that it uses Performance Explorer trace under the covers for the data collection; however, job trace does much more than just provide a customized interface over Performance Explorer.
• Performance Job Trace
  Start Job Trace (STRJOBTRC) is a job trace function to collect performance statistics. As of the 6.1 release, it invokes STRTRC.
• Performance Explorer
  Performance Explorer trace (STRPEX) is used to collect very detailed data, primarily intended for performance diagnostics.
• Licensed Internal Code (LIC) traces
  LIC traces can be started by going into System Service Tools (SST) or Dedicated Service Tools (DST) and starting the LIC trace service function. The Trace Internal (TRCINT) command is a command interface to LIC trace that alleviates the need to go into SST or DST to start or manage your LIC traces.
• Trace Connection
  The Trace Connection (TRCCNN) command, which provides a trace similar to a communications trace, actually does a LIC trace under the covers, so TRCCNN will start a TCP/IP or Sockets LIC trace.
• Performance Trace
  Start Performance Trace (STRPFRTRC) collects multiprogramming level (MPL) and transaction LIC trace data. It actually invokes TRCINT to start a LIC trace.
• Communications Trace
  Communications Trace (STRCMNTRC) is used to collect trace data as it flows over communications lines.
• Trace TCP Application
  Trace TCP Application (TRCTCPAPP) is a specialized interface for starting traces specifically for a set of TCP/IP Application Servers. Of interest is that you can also start additional traces when you trace a TCP/IP application server, such as job traces or LIC traces.

As you can see, there are a wide variety of interfaces to collect trace data, but it really comes down to four basic types of trace data collection:

• Job Trace
• Performance Explorer
• LIC Trace
• Communications Trace

The Work with Traces command can allow you to manage any one (or all) of these traces. It has one parameter – the Trace (TRC) parameter – which allows you to specify which traces you want to work with. Supported values are:

• *ALL – work with all traces
• *CMN – work with communications traces
• *JOB – work with job traces
• *LIC – work with LIC traces
• *PEX – work with Performance Explorer traces

When you first go into the WRKTRC interface, you will see the traces that are applicable based upon the TRC parameter. If there are no traces active, a message will be displayed stating that no traces are found. What's particularly useful with WRKTRC *ALL is it will show you all the traces that exist on the system in some form (active or ended); it is not uncommon to start a trace but forget that it is running; WRKTRC can help you determine if you have forgotten traces that are still running. Or perhaps you have a trace you started and ended, but never deleted (to free up storage); WRKTRC will show you those traces.

The F6 key allows you to start a new trace. You can then select the type of trace you want from the menu options listed. The appropriate command prompt interface for starting your selected trace is then displayed.

In the following screen capture, you will see an example of working with traces with one trace of each type active:

 

 

The following screen capture is an example of working with traces when there is a trace of each type that has been ended:

 

 

For any trace, you can take actions that are appropriate for that trace type and the state that it is in. Details of what actions are appropriate for what trace types can be found in the online help. The online help also describes the various statuses that can occur for each type of trace.

Finally, there are some additional diagnostic trace capabilities that exist on i that are not supported with WRKTRC; this is because these trace types are not commonly used any more. For example, Trace CPI Communications (TRCCPIC) and Trace ICF (TRCICF) are related to SNA communications functions. User Trace (xxxUSRTRC) is replaced with Job Trace.