Performance Reports With the Performance Data Investigator

Several weeks ago I wrote about the latest enhancements to the Performance Data Investigator in Performance Data Investigator – Better than Ever.

That article had the following paragraph:

Performance Reports
You now have the ability to generate performance reports from PDI. In the past, you could get text-based reports from the PT1 product, but these were difficult to analyze. The performance reports provide graphical views of the performance data and can be customized to report on the metrics that are important to you.

Note that this reporting feature is part of the latest Navigator for i Enhancements and you need to install the PTFs documented in that blog.

Let’s assume your manager wants a weekly update on the some key performance metrics for your IBM i partition from your peak timeframe. In this contrived scenario, we’ll say that is every Friday – we’ll pretend Friday is the busiest day for your business and there are some critical jobs that run each Friday night. Every Monday morning, your manager wants to review CPU utilization, page fault informationand disk performance information from that timeframe (he’s particularly concerned about several disk performance metrics since there have been ongoing concerns about disk performance). With the new reporting feature, you can easily create a report definition that contains the desired metrics. You can then generate that report every Monday morning by running the report over the Collection Services data from the critical timeframe of the prior Friday.

Let’s walk through how you do this. 

Reports are found by expanding Performance>All Tasks>Reports>Performance Data Report Definitions.

You will find three predefined report definitions. You can create your own report definition to get exactly the report you want.

The report definition is separate from the report itself. The report definition specifies what perspectives (a perspective is simply a chart or table) you want to include in the report. You can then run that report definition over your selected Collection Services collection to create the desired report.

For our scenario, we will create our own custom report definition that includes the metrics we are interested in. From the report definitions page, you select Actions>Add Performance Data Report Definition. This will bring up the page where you define your report. You give it a name and description, then use the Add button to chose the performance charts (perspectives) that will be contained within your report. When you are adding a perspective you get the same navigation tree you see when you use the Investigate Data task. You simply select the charts or tables you want included in your report.

In my example, we are using Collection Services data and we want the following charts:

1. CPU Utilization and Waits Overview
2. Page Faults>Page Faults Overview
3. Disk>Disk Response Time>Detailed>Disk I/O Rates Overview - Detailed

For the Collection specified on the definition, we will let this default to the most recent collection in the default collection library (which is usually QPFRDATA). This is the collection that will be used by default when you create the report; you can override this setting when you create the report.

In the Cover Page field, you can add whatever text you want displayed on the first page of the PDF.

The report definition properties look like the following:

After you have created your report definition, you can then run that report over the desired performance collection, using the Actions>Create Performance Data Report, and when you do this, you have the ability to specify the collection over which the report will be run. In our contrived example, we would create the performance data report each Monday morning, selecting the Collection Services collection from the prior Friday.

 

 

In my example, I chose to generate my report as a PDF, which I can then share with others.

This reporting feature is extremely easy to use and very flexible. And you have it – it’s all part of Navigator for i; there is nothing for you to do except to make sure you have installed the latest PTFs required for Navigator.

Start using it today!

 

 

 

 

IBM i Performance Frequently Asked Questions

IBM recently published an IBM i Performance FAQ. This document is a fantastic resource on all types of performance topics. It reviews the various components of system performance, discusses benchmarking, sizing and capacity planning, monitoring, and covers the performance tools that are available on i. The document has answers to many commonly asked questions and ends with an extensive list of references.

The IBM i Performance FAQ was put together by the Systems Performance team in Rochester, drawing upon the knowledge of leading experts on IBM i performance from the development team, the support center and Lab Services.

All IBM i administrators should read this document and it should be used as an initial reference for any performance-related questions. It is the single best place to start for anything related to performance topics on i.

The Performance management on IBM i Web page has also been updated to include a link to this performance FAQ. If you’re not familiar with the performance management site, it’s another great resource for information on performance topics.

Finally, the End-to-End Performance Management on IBM i Redbooks publication is a good reference for IBM i performance tools. Although it was written based upon IBM i 6.1 capabilities in 2009, the content of that Redbooks document remains current.

 

 

Commands to Manage Performance Collections

A few weeks ago I wrote about the various Performance Data Collectors in IBM i. It’s great to be able to collect such a wide variety of performance data, but what do you do when you want to manage that data? Maybe you have the need to do some analysis on a particular performance data collection, but you do not want to run the analysis tools (which have overhead) on your production partition. How do you move that collection to a different partition? Or maybe you are trying to develop a strategy for how to save (and possibly restore) your performance data; you never know when you might need a performance collection from a critical time period for comparison purposes if something goes wrong. Perhaps you simply want an easy way to delete some performance data that you no longer need.

Consider what a performance collection consists of: it may be data in a management collection object or it could be data in a set of database files. When the data is in the database files, each type of performance collection has its own set of files: 

• QAPMxxx for Collection Services
• QAPYJWxxx for Job Watcher
• QAPYDWxxx for Disk Watcher
• QAYPExxx for Performance Explorer

Each collection consists of a member with that collection name in each of the files for that collection type. Prior to 6.1, it was rather difficult to manage an individual collection of performance data.

In the 6.1 release, in addition to adding the Job Watcher and Disk Watcher performance collectors to the operating system, IBM also made it much easier to manage all of your performance data. The blog, i Can Find All my Performance Collections in One Place, described the Collection Manager – the graphical user interface for managing all of your performance data collections.

That blog made brief mention of the capability to copy, delete, save and restore performance data through the GUI.

The support behind the GUI is actually a set of commands that were added to the 6.1 release. These commands are: 

• Copy Performance Collection (CPYPFRCOL)
• Delete Performance Collection (DLTPFRCOL)
• Restore Performance Collection (RSTPFRCOL)
• Save Performance Collection (SAVPFRCOL)

All of these commands have a very similar interface and all of these commands can be used to manage any of the types of i performance data. You have to know the name of the performance collection you want to work with and the library in which it resides; you also have to specify the type of performance data you want to manage. You can select from the following (the default is *CSFILE):

• *CSFILE
  Collection Services file-based collections

• *CSMGTCOL
  Collection Services object-based collections

• *DWFILE
  Disk Watcher file-based collections

• *JWFILE
  Job Watcher file-based collections

• *PEXFILE
  Performance Explorer file-based collections

• *PEXMGTCOL
  Performance Explorer object-based collections

• *ALL
  All types of performance collections. This includes file-based collections and object-based collections. 

A bit of explanation is in order for “file-based” collections versus “object-based” collections.

When using Collection Services and Performance Explorer, data is initially deposited into a high-performing, highly scalability repository called a Management Collection object. The performance data can also be copied into a set of database files, which is required for analysis. The data in the management collection object is referred to as an “object-based” collection, while the data that has been copied into the database files is referred to as a “file-based” collection. You will note that for Disk Watcher and Job Watcher, no intermediate repository is used (the data is deposited directly into the database files), and as such, those types of collections only have options for “file-based” collections.

 

 

 

Performance Data Investigator

A few weeks ago I wrote about the Performance Data Collectors in IBM i; you can collect many different kinds of performance data, but once you have the data, what is the best way to view and analyze that data? Use the Performance Data Investigator. If you are on 6.1 or 7.1 you already have it.

The Performance Data Investigator was introduced with the 6.1 release and is found in the Navigator for i web console; it is not available in System i Navigator (and it will never be in System i Navigator).

Within the Navigator browser console, you will find the Performance tasks in the left-hand navigation pane. When you select the Performance tasks, you will see the following options:

The Investigate Data option is how you access the Performance Data Investigator. When you take the Investigate Data option, you will see something like the following image:

The initial list of perspectives shows you the set of content packages that are available; a content package is simply a grouping of perspectives for the same type of performance data. The Collection Services content package contains all the graphs and tables for Collection Services data. Which content packages you see depends upon what options of the Performance Tools licensed program product are installed.

You will always see Collection Services and Health Indicators; the capability to view and analyze Collection Services data is part of the base operating system and no other product is necessary.

However, if you want to analyze Disk Watcher or Performance Explorer data, you will need to have the Performance Tools product, Manager feature installed. To analyze Job Watcher data, you need to have the Performance Tools product, Job Watcher feature installed. Below is a screen capture of the Performance Tools product with all the options installed:

On the main Investigate Data panel, open the type of performance data you want to analyze – for example, you can expand the Collection Services folder to show the following navigation:

You can see that the list of perspectives (which is simply a fancy name for a graph or table) is organized by the type of information you want to see. You can further expand the folders to find the perspective you want to view. Once you have identified the view you want to see, you can display it.

Note at the bottom of the screen you have the capability to select which performance data collection you want to display, so you can have various performance data collections in different libraries.

Once you have identified the graph or table you want to display along with the performance collection you want to use, you display the data. An example of what a graph looks like is below:

The Performance Data Investigator has a wealth of capabilities; for a more complete article on the Performance Data Investigator, please refer to the IBM i Performance Data Investigator article on the developerWorks site.

Finally, here is a list of prior blogs with information related to the Performance Data Investigator:

Investigate Performance Data

Display CPU Utilization for all Partitions

Tell You Why You Are Waiting

Understand Scaled CPU Time

Measure Disk Response Times

Customizing a Perspective in PDI

Find All My Performance Collections in One Place

Job Performance Information

Performance Data Collectors in IBM i

 

 

 

Performance Data Collectors in IBM i

There were significant changes to the IBM i performance tools in the 6.1 release. The ability to collect additional types of performance data, management of this data, and a graphical interface for viewing and analyzing this data were added. I often speak on performance topics, but I have not written much about them. Since there is much to cover, this will be the first in a series of articles. A good place to start is to review the performance data collectors available on i.

Beginning with the 6.1 release, there are four different types of performance data you can collect; two of them newly integrated into the operating system with that release. IBM i has the richest set of performance data compared to any other operating system; I believe it is a substantial differentiator, giving i users greater insight into how their machines are functioning.

 

Collection Services

Collection Services has been part of the operating system for many releases. Most administrators should be aware of it and using it regularly. Collection Services collects performance data from throughout the operating system on an interval basis and is designed to be running continuously. The overhead for data collection is not significant, but you do need to carefully manage the data that is collected. The major change with Collection Services in the 6.1 release is that it was changed to always be running by default.

Collection Services is used for day-to-day management of your system's performance; you use it for your baseline performance data collection to more quickly determine if changes have occurred that affect performance. It's the underlying data for Management Central System Monitors, it's the data that is sent to IBM if you use PM for Power Systems, and it's also the data that's used for trending and capacity planning purposes.

 

Performance Explorer

Performance Explorer, commonly referred to as PEX, has been part of the operating system since the V3R6 release. PEX has three modes of operation and can collect statistical performance data as well as trace data. Today, PEX is generally used for two reasons: 1) as a performance collector to aid with detailed application performance analysis, and 2) to collect detailed performance for diagnostics on the most difficult performance problems. PEX used to be used much more, but the development of the Job Watcher performance data collector has changed that.

 

Job Watcher

Job Watcher first appeared around the V5R1 time frame. Job Watcher was initially developed by the Rochester Support Center as a tool to aid performance diagnostics. Job Watcher and the graphical user interface used to analyze the data proved to be very powerful for performance diagnostics. The iDoctor offering was the first place where Job Watcher was made available to our clients. Since Job Watcher was so effective at performance diagnostics, it was built into the operating system with the 6.1 release.

Job watcher can collect much of the same information that Collection Services does, but it collects much more; call stacks, SQL statements, wait information (see i Can ...Tell You Why You Are Waiting for more information about waits), activation group information, Java information and more. In addition, while Job Watcher uses a sampling technology similar to that of Collection Services, Job Watcher has the option to sample as fast as possible, which can provide data similar to a trace.

Job Watcher is the most commonly used performance data for detailed diagnostics. The Rochester Support Center performance analysts, as well as the performance consultants in IBM Lab Services generally use Job Watcher. Nearly every performance problem can be resolved with Job Watcher data.

 

Disk Watcher

Disk Watcher is a specialized performance data collector that collects data specifically for disk operations. Its use is intended for detailed diagnostics of disk performance problems or for the use in optimizing disk I/O in applications.

All of these performance data collectors have a set of commands that you can use to define the data you want to collect, commands to start and end the data collection, and commands to manage the data. There are graphical user interfaces that I'll cover in the future as well.

A prior blog article on Find All my Performance Collections in One Place reviews the Collections task in the Navigator for i Web console. The Collections task can be used to see all of your performance collections, regardless of what type, in one consolidated interface.

As I noted at the beginning of this article, IBM i's performance data collectors are second to none; the combination of Collection Services for day-to-day performance management and Job Watcher for detailed diagnostics provide a powerful combination of tooling that can’t be beat.

 

 

 

Job Performance Information

Last week I wrote about the basic performance information that had been added to the CPF1240/CPF1241 messages, and in that blog also mentioned the basic performance information in the CPF1164 message that is sent to the history log.

As reader Tim Hawkins commented:

If you really want granular performance information (without the overhead of the Performance Tools assuming you have it), use the Job Accounting journals, there are several tools available to convert them to DB files. Then query the data using your favorite tool.

Tim, not realizing it, was the “straight-man” for this week's blog; although I didn't state it, when I was writing last week's blog, I was planning to write next about job performance data, the options available, and when you might chose which option.

IBM i has very rich features regarding performance metrics; while the capabilities are very comprehensive, it can sometimes be confusing as to what feature you should use when. Of course, the answer to every question related to performance is “It depends!” I've not written about much of these capabilities since I speak on them frequently, but you can be sure they are on my list of future topics.

This week, I'll review the basic options for job performance data and provide brief guidance for when you might consider each option.

• CPF1164, CPF1240/CPF1241 messages as reviewed last week.
  Active Jobs, Server Jobs, System Status, etc...

The commonly used interfaces used such as work with active jobs provide basic job performance information in an interactive interface (either via the command line interface or the graphical user interface). The GUI also has the ability to look at server jobs as a subset of active jobs. These interfaces are useful for a quick view of performance across a small set of jobs.

There are also API interfaces to retrieve this information programmatically.

• IBM i Job Notification Exit Point
  The Job Notification exit point can be used to log notification messages to data queues when IBM i jobs are placed on a job queue, a job starts, or a job ends.
  With the job notification exit point, when a job ends, you will have the date and time the job entered the system, when the job started, and when the job ended. You can also get the processing time used; no other performance data is available.
  
  The job notification exit point would be useful if all you need is awareness of jobs starting and ending, and possibly the processing time used.

• IBM i Job Accounting
  The When to Use Job Accounting article in the Information Center has a nice summarization of when you'd use job accounting data as opposed to when you would use the data in the CPF1164/CPF1240/CPF1241 messages.
  
  Job accounting data includes all the data in the above-listed messages. Job Accounting data includes additional metrics, such as print and communications information (note the communications information is for SNA communications).
  
  You would use job accounting if print and communications information is important to you or the ability to assign accounting codes is required.
  
  Also, the performance data in the CPF1164 message pre-dates the job accounting information and job accounting would be the preferred way to gather this type of job data today. If IBM were to add additional, basic performance metrics, it would be added to the job accounting data and not to the messages. IBM i developers are aware there are customers still using the old method of getting the basic performance data from the messages in QHST so it will continue to be supported.

• The Job Accounting Journal entry documentation details the information that is collected.
  Collection Services Performance data
  
  Collection Services, which runs 24x7 by default starting with the 6.1 release, has comprehensive job performance data collected. This data is stored in the QAPMJOBMI and QAPMJOBOS database files. In addition to the job performance metrics, Collection Services also collects save/restore metrics for a job in the QAPMJOBSR file. Finally, wait information is also collected by Collection Services and provides the ability to determine Why you are Waiting.
  
  If you need low-overhead performance data collection with detailed performance data, Collection Services provides the best solution; when you run Collection Services, it will collect and save detailed job performance data before you know you need it! You can query the data from the files directly, or you can use the Performance Data Investigator to graphically Investigate Collection Services data.

• PM for Power Systems
  You probably are familiar with PM for Power Systems from the “old” name of PM/400. PM for Power Systems summarizes data from Collection Services and optionally sends that summarized data to IBM. By choosing to use PM for Power Systems, you can get summarized trend analysis information on your system's utilization and performance, plus job information (most frequent, processor time, I/O and print) on the top 10 low priority jobs. Remaining capacity is constantly calculated and the data are readily available to the IBM Workload Estimator for sizing future needs.

• Job Watcher Performance data
  The Job Watcher performance data collector was added to the operating system in the 6.1 release. It provides the ability to get detailed job performance data and collects much of the same job data as Collection Services does, but it also collects additional information, such as call stacks, SQL statements, objects being waited on, and more.
  
  Job Watcher is used for detailed job performance analysis and diagnostics. You can view Job Watcher data with the Performance Data Investigator or the iDoctor performance tool.
  
  

As you can see, there are a number of alternatives available. Which one you chose will depend upon the specific requirements you have for why you need job performance information.

CPF1240 and CPF1241 Messages Contain Summary Performance Data

You probably know that the CPF1164 (Job Ended) message that is sent to the history log contains some basic performance information regarding the job that ended. You can't see the details of this information via the Display Message Description (DSPMSGD) command or via the Display Log (DSPLOG) command, but the data is available in the history log files and you can write application programs to retrieve this data.

The basic performance information on the CPF1164 message includes:

• Date and time the job entered the system
• Date and time the job started
• Total response time (for interactive jobs)
• Total transactions (for interactive jobs)
• Number of synchronous auxiliary I/O operations

The Performance Information and QHST section in the Information Center is where documentation on this can be found.

What you may not be aware of, is in the IBM i 6.1 release, basic performance information was also added to the CPF1240 (Job ended abnormally) and CPF1241 (Job completed normally) messages.

Unlike the CPF1164 message, the performance information on the CPF1240 and CPF1241 messages is part of the message description and you can display the information with the DSPMSGD command. For example, a DSPMSGD CPF1240 will show the following information at the end of the technical description information:

   -- Entered system date and time:      &9

   -- Started date and time:             &10

   -- Job type:                          &16

   -- Job subtype:                       &17

   -- CPU seconds used:                  &11

   -- Synchronous auxiliary I/O:         &12

   -- Asynchronous auxiliary I/O:        &13

   -- Ending code:                       &14

   -- Secondary ending code:             &15

And by looking at the field data of the message, you can find the details of the data type and length for each of the replacement variables.

There are a few minor differences from the data that is included with the CPF1164 command.

• The total response time and total transactions fields are not part of the information on the CPF1240 and CPF1241 messages; these messages are only sent for batch jobs and total response time and total transactions are only applicable to interactive jobs.
• Asynchronous auxiliary I/O counts are included on the CPF1240 and CPF1241 messages.

i Can… Find All my Performance Collections in One Place

This week's blog was written by Lora Powell. Lora works on the IBM i Performance Tools team and is the lead for the Performance Tasks GUI in IBM Systems Director Navigator. Thanks, Lora.

If you are new to 6.1 or 7.1, you may not be aware that you can go to one place to see and work with all your performance collections.

When working with Performance Collections on 6.1 and beyond, some changes were made from previous releases. Along with the change in Performance Collection Tools to collect Job Watcher, Disk Watcher, Collection Services, and Performance Explorer data by the base operating system, an internal collection table was added to keep track of these collections on the system.

The Collections task gives you access into this table. To view and work with all the collections on a system, start with IBM Systems Director Navigator for i (hint: to start this, simply type in your Internet Explorer or Mozilla Firefox browser the following URL: http://systemName:2001 where “systemName” is replaced with the name of your system). Select Performance on the left-hand navigation pane. Then select the Collections task.

 

This brings up the Collection Manager list of all the collections found on your system, from all performance collectors, including both the management collection objects (*MGTCOL) as well as generated database files.

Here you can work with your collections using the GUI interfaces to the Copy, Delete, Save and Restore commands. Selecting Properties will display general and detail information about the collection. You can select action Create Data on a management collection object to create the collection services file based collection. You can also Convert performance data from a previous release to the format needed for processing by the current release. Selecting Investigate Data on a collection services file based collection, disk watcher, job watcher or PEX collection will bring up a default perspective displaying the data from the collection with one or more charts and tables.

The collection table (located in QUSRSYS/QAPMCCCNTB) is updated whenever new collections are generated, restored or deleted from the system using the performance collection tools commands: CRTPFRCOL, RSTPFRCOL, or DLTPFRCOL (xxxPFRCOL), or the GUI equivalents as discussed above. If however, a collection was restored from another system using the RSTOBJ or RSTLIB commands, or if the collector job was ended abnormally either due to system or subsystem ending immediately, the table will not be updated accordingly.

Likewise, if you use DLTLIB to remove a collection, the collection table will not be updated. The table information for collections will then be inconsistent with the actual collections that exist on the system.

If any of the above situations occur, you may notice this inconsistency when using Performance Data Investigator (Investigate Data task under Performance). The list in the collection name dropdown is populated by the collections found in the collection table, so it will only show the collections from the table.

The collection table can be rebuilt at any time to update it with the libraries and collections that exist on the system. To rebuild the table, from the Collections panel, use Select Action>Rebuild Collection Table.

This may take a few minutes. When complete, click on the Refresh button on the Collections panel. The Collection Manager list will now include any previously missing libraries and collections, and any collections that were deleted will no longer be shown.

To then use Performance Data Investigator with the updated collection table, close any previously opened Investigate Data tabs, and go back in to Investigate Data. The Collection Library dropdown will now be populated with all the collections that currently exist on the system.

Note that with the latest PTFs installed, you will see that the Collection Name dropdown list also shows the date and time that the collection was started. The default name of the collections generated by Collection Services is ‘Q’ plus the Julian data and time. Since this is difficult for most to translate into a “real” date, adding the human-readable format of the date and time stamp makes it easier to find the collection you are looking for.

For the latest PTF information affecting the Collection Manager and Performance Data Investigator, check out the developerWorks Technology Updates page recently launched under the Performance Tools category: https://www.ibm.com/developerworks/ibmi/techupdates/perftools.

 

 

Lesser-Known Features of Work With System Activity

Hopefully most of you now know that in the 6.1 release, IBM moved the Work with System Activity (WRKSYSACT) command from the Performance Tools licensed program product into the base operating system. WRKSYSACT has been around for a long time, but it has a few features that you may not be aware of.

Work with System Activity is a great tool for getting a basic understanding of what is happening on your partition in terms of performance. WRKSYSACT will show you jobs, threads, and tasks that are using system resources – CPU utilization, synchronous and asynchronous I/O, storage allocation information, etc. WRKSYSACT is the only green-screen command interface in the operating system where you will find information about tasks. Other command interfaces – such as WRKACTJOB – only show information about jobs and threads.

It would take far too much space to write about all of the things you can see through WRKSYSACT; rather, there are a few features that I want to discuss in more detail.

 

WRKSYSACT in Batch Mode

First of all, did you know that you can run WRKSYSACT in a batch monitoring mode and write the performance data to a file? You enable the batch mode with the OUTPUT parameter; by default this is * and the information is presented on your display. But there are two additional options, *FILE and *BOTH, which will write the performance data to a database file (QAITMON in QPFRDATA by default). You can specify the sample intervals with the INTERVAL parameter (which also controls the auto-refresh intervals as well) and how many intervals are collected (NBRITV parameter). When you take the option to write the data to a file, a job named WRKSYSACT is submitted to QBATCH to collect the performance data.

The Print Activity Report (PRTACTRPT) command can then be used to print out a report from the sample data collected by WRKSYSACT. The PRTACTRPT command can generate a summary report as well as a detailed report and has options to allow you to specify a timeframe of interest, specific jobs you may want in the report, as well as what criteria the report should use in ordering.

 

Wait Information

Wait information will tell you what a particular job, thread or task is waiting on when it is waiting to do more work. I wrote about wait accounting in “i Can Tell You Why You're Waiting” some time ago. You can also see wait information using WRKSYSACT; you simply need to determine which job/thread/task you want to look at, and use option 6, Wait detail, for that job or task. Option 6 will bring up a display that will list the wait categories for that job/thread/task. Be aware that this is a high-level summary of the kinds of wait for that job or task. However, it make give you some clues about what job or task you need more detailed information for; you can then use the Performance Data Investigator with Collection Services or Job Watcher data to view detailed wait information.

 

Storage Allocated/Deallocated

WRKSYSACT will also show the storage allocated or deallocated by a job/thread/task by pressing the F11 key to get to View 4. I wrote about “Understanding Disk Space Usage” and in that blog reviewed WRKSYSACT and the storage allocated/deallocated information.

 

Scaled CPU

Scaled CPU time is a metric that can be used to determine if energy management features, such as power capping or power savings, are in effect on your IBM i partition. These energy management capabilities can be turned on through Active Energy Manager, which is a plug-in to IBM Systems Director. When you use these energy management features, the system's processor may run at lower or higher frequencies, which can affect your partition's performance. I wrote about this topic at length in the blog “i Can Understand Scaled CPU Time.”

On the WRKSYSACT display, the value “Average CPU rate” will be displayed if you are using energy management capabilities. This rate is an easy way to determine if your processor has been slowed down due to power capping or power savings.

One final note – one thing that has not changed with WRKSYSACT is the limit of one user at any given time. WRKSYSACT collects a lot of information from critical interfaces in the operating system and the limit of one user ensures limited contention on those critical interfaces.

 

 

 

 

IBM i Diagnostic Tools – i Does it Better

A few weeks ago, I wrote about Job Trace, followed by an update on how you can instrument your applications for IBM i job trace. Job trace is but one of many tools used to collect diagnostic data on i.

The diagnostic tools integrated within the operating system provide the capability to collect detailed data that's required identify and resolve the root cause of software problems. The tools used for problem determination fall into two main categories -- the tools used for functional problems and the tools used for performance problems.

Tools used to resolve function problems -- whether the problem might be incorrect results, errors and exceptions, or unexpected failures -- are generally tools that support dumps, logs, and traces. On i, logs are generally in the form of messages; job logs, message queues and the history log are well known by i users.

Traces can be collected at the operating system level (aka, “above the MI”) or at the Licensed Internal Code level (aka, “below the MI”). There are different kinds of traces that are used depending upon what part of the system needs to be traced -- job trace, LIC trace, communications trace, trace TCP application and many more. “Flight Recorders” are a special always on type of trace that can be dumped out when necessary to start problem determination without recreating the issue.

Dumps are methods to collect detailed data, whether that data is for a job, an object or a data structure. Dumps may be used to collect internal information as well as application data details. Dumps can be initiated from commands and APIs, as well as the Display/Alter/Dump service function.

Performance tools provide the capabilities used to aid with the root-cause resolution of a performance problem; performance tools are generally used to identify why something isn't performing as it should, optimizing and tuning performance to keep it running well, and planning for future growth. The i performance tools tend to collect data at all the various layers of the system (the OS and LIC), so different tools are not required depending upon where the problem is suspected to be. The IBM i performance tools include Collection Services, Job Watcher, Disk Watcher and Performance Explorer. A future blog article will explore the different kinds of performance data collectors and when you would use each type.

The following simple diagram shows the primary diagnostic tools for functional or performance diagnostics and whether the tool is used for operating system or LIC diagnostics.Perhaps one of the biggest challenges is understanding all the tools that are available and when to use each one -- after all, there is a very rich set of diagnostic tools built into the operating system.

The second challenge is capturing the data needed for root cause problem resolution -- often times the required traces are not running the first time a problem occurs and the problem must be recreated with additional diagnostic tools enabled to required debug information is collected. That's why some of the capabilities built into the operating systems are always on, allowing for some level of troubleshooting without the need to recreate the problem. Messages, flight recorders, Service Monitor and Collection Services are all examples of features that IBM intends to have running all the time to enable this “First Failure Data Collection” -- the ability have the detailed information to begin diagnostic sleuthing the first time a problem occurs.

This rich set of diagnostic tools is a differentiator for IBM i; everything you need for diagnostics is built into the operating system. Analysis of that data, however, might present other challenges! On other platforms, diagnostic data is often simply written to flat files (called stream files on i) and are often just text strings, with little consistency from one piece of software to another on how the diagnostic data is collected. The development team for the i operating system has guidelines on how code should be instrumented for functional and performance diagnostics and the consistency these guidelines provide is a major value-add of i.

Two excellent Redbooks resources cover the diagnostic tools for i:

• i5/OS Diagnostic Tools for System Administrators: An A to Z Reference for Problem Determination 
  This redbook was written with the V5R4 release, but the information is still applicable to 6.1 and 7.1.
• End-to-End Performance Management on IBM i
  This redbook covers everything you might want to know about performance management on i, including all of the performance tools that are built into the system and the basics of when and how to use each.