i Can … Automate Monitoring With Watches

In V5R3, IBM i introduced watches as a way to automate the ending of traces. In 5.4, watches were supported independent of the trace commands.

Watches provide a way to automate tasks when certain events occur. An event can be a message, a Licensed Internal Code (LIC) log (also known as a VLOG), or, in 6.1, a Problem Activity Log (PAL) entry. The primary motivation for adding watches to the operating system was initially to provide a way for improved diagnostics, but watches, particularly message watches, can be used for automated monitoring of system conditions. Watches provide a way to be notified programmatically when the event occurs so immediate action can be taken. Additionally, watches can be very useful at detecting situations that occur intermittently since the actions can be automated.

Watches have minimal system overhead when they’re defined, but not “hit.” When the watch condition occurs, the actions taken depend upon the how the watch is defined. When using watches independent of traces, the actions taken are what the watch exit program is coded to do.

To start a watch, use the Start Watch (STRWCH) command or QSCSWCH API. To end a watch, use the End Watch (ENDWCH) command or QSCEWCH API. You can also use a Work With Watches (WRKWCH) command to display and set actions on watches. Up to 10,000 watches can be active at one time.

Watching for LIC logs or PAL entries is probably something you won't need to do, but watching for messages can be an effective way to perform proactive and automated monitoring of system conditions. You can watch for messages sent to any message queue, the history log or job logs. When you define the watch condition, you can specify several different conditions you want to check for to limit the situations under which the watch will trigger. When the watch condition is matched, your exit program will get control and you can take whatever actions you deem appropriate for the condition.

Using watches to automate monitoring of messages is much more efficient than using the Management Central Message Monitors. Management Central Message Monitors use a polling technique to retrieve messages, which has more overhead and is less timely than watches. With Management Central Job Monitors, you can monitor for job log messages, however, monitoring job log messages with Management Central Job Monitors is expensive in terms of the system resources used due to its polling nature. Message watches are a great alternative since they provide essentially the same support with significantly fewer system resources used. However, it isn’t as easy to set up Message Watches since there’s no GUI for watches; only command and API interfaces have one. In addition, with watches you have to write your own watch exit program, whereas Management Central has a nice interface to define the actions taken when the monitored condition occurs. IBM has provided an example watch exit program in the Information Center.

When watches are defined, additional jobs run on the system. These jobs are different on the various releases. On 5.4, there’s one batch job per watch in the QUSRWRK subsystem. These jobs run until the watch is ended. In 6.1, watches were changed to use batch prestart jobs. There’s a single batch job, QSCWCHMS, in QUSRWRK for message watches; when a message watch condition is hit, the user exit program is called in a prestart batch job, QSCWCHPS. This change in 6.1 reduces the number of jobs in the QUSRWRK subsystem.

IBM i has a function called Service Monitor that uses watches as its notification mechanism. Service Monitor was added in the 5.4 release. Service Monitor is controlled by the QSFWERRLOG system value; when this system value is set to *LOG, service monitor will be active. Service Monitor uses many watches, and you’ll see evidence of this by the jobs running in QUSRWRK, as well as when you work with watches. I'll talk more about Service Monitor in a future blog article.

 

 

i Can .... Display CPU Utilization for all Partitions

In last week's blog, I mentioned that with the latest Performance Data Investigator enhancements, IBM i now provides physical system charts that display the overall CPU utilization for the physical box and all partitions, regardless of operating system.

This is an exciting enhancement that many of our customers have asked for, so I want to tell you more about it and how you enable this feature.

Many IBM i customers are using logical partitioning with their systems. When using multiple partitions, many customers want to understand the overall utilization of their processing capability, across all partitions, regardless of whether the partition is running IBM i, AIX or Linux. To do this, many have created their own customized solutions by querying Collection Services data from each partition and pulling together an overall view of CPU utilization; but if you have AIX or Linux partitions, there’s missing information unless you have the skills to collect and understand the performance data from those partitions as well.

With POWER6 hardware and IBM i 6.1, you’ve been able to collect CPU utilization information for the physical system since firmware level xx340_061 was introduced late last year. However, that just enabled the collection of the data; IBM provided no way to view it. With the updates to the Performance Data Investigator I reviewed last week, additional charts are now available for Physical System views of CPU utilization that now let you easily view this data.

However, there are some specific configuration requirements. As I’ve already said, you must be running on POWER6 hardware with a minimum firmware level of xx340_061. You also must be using the IBM i 6.1 release. Then, you must also enable the collection of this performance data for the desired partition. You only need to collect this data on one partition and it must be an IBM i partition. The CPU utilization information collected will reflect work done in partitions running AIX and Linux as well as IBM i, but AIX and Linux don’t support collecting this data. Only IBM i has this data collection capability.

To enable the collection of this performance data requires the setting of a configuration parameter on the HMC or Integrated Virtualization Manager (IVM). On the HMC, there’s an "Allow performance information collection" checkbox on the processor configuration tab. Select this checkbox on the IBM i partition that you want to collect this data. It's not quite as simple with IVM; if you’re using IVM, you use the lssyscfg command, specifying the all_perf_collection (permission for the partition to retrieve shared processor pool utilization) parameter. Valid values for the parameter are 0, do not allow authority (the default) and 1, allow authority.

Once the performance data collection support is enabled, Collection Services will collect this additional information. At each collection interval, Collection Services will collect partition configuration and utilization information from the hypervisor; the data is stored in a file named QAPMLPARH. 

The latest update to the Performance Data Investigator includes additional perspectives in the Collection Services content package for this data in a folder called "Physical System." It also includes the following charts:

• Logical Partitions Overview
• Donated Processor Time by Logical Partition
• Uncapped Processor Time Used by Logical Partition
• Physical Shared Processor Pool Utilization
• Physical Processors Utilization by Physical Processor
• Dedicated Processor Utilization by Logical Partition
• Physical Processors Utilization by Processor Status Overview
• Physical Processor Utilization by Processor Status Detail

Here’s an example of the Logical Partitions Overview chart, which shows configuration data and CPU utilization for all logical partitions on the system:

 

If you’re on POWER6 hardware, but have not yet installed IBM i 6.1, you may want to think about creating a small i partition and experimenting with this new capability to manage your machine's processor utilization across all partitions.

 

i Can ... Display Call Stacks

IBM i has always had the capability to display the call stack associated with a job. When threads were introduced on the system, call stacks were provided on a per-thread basis.

In V5R4, IBM made a significant enhancement to expand the information that's displayed when viewing a call stack. Specifically, to provide improved integration and support for Java and Portable Application Solutions Environment (PASE) applications, the call stack support was enhanced to display call stack entries for Java applications, as well as applications running in PASE. The capability to display Licensed Internal Code (LIC) stack entries was also added.

You can view call stacks from the Work With Job and Display Job commands; option 11 will display the call stack of the initial thread in the job. If the job has secondary threads, you must first select the option to work with threads, from which you can find the option to display the call stack for a specific thread. In addition, the graphical user interfaces (Navigator for i and IBM Systems Director Navigator) also support viewing call stacks. To view PASE kernel or LIC call stack entries, you must have *SERVICE special authority and you need to select the option to "include" the additional call stack information. On the green-screen interface, this option is F21. On the graphical user interfaces, you can use the action drop-down menu to select the include option to enable the additional call stack information.

The type of call stack entry is identified in the "Type" column. On the graphical user interfaces, the type column isn't displayed by default; you need to go to the "columns" dialog and select it. When it's displayed, it's a text description of the type of call stack entry. On the green-screen interface, the type column is abbreviated. Pressing F11 on the green-screen interface provides additional information on the call stack panels, and the third view provides an extended type column that expands upon the basic type field. You may see the following type and extended type fields:

   Blank - an OPM or ILE invocation

   001-999, +++ - the OPM or ILE request processor level

   L - Licensed Internal Code

      *LIC

   P -  PASE user code

      *PASE_32 - PASE 32-bit

      *PASE_64 - PASE 64-bit

   K - Pase kernel code

      *PASE_KERNEL

   J - Java code

      *JAVA_JIT - Java Just-in-time

      *JAVA_MMI - Java Mixed Mode Interpreter

      *JAVA_INTERPRET - Java Interpreted

      *JAVA_GLUE - Java glue frame

In addition to the support through work management interfaces for displaying call stacks interactively, you can also get call stack information in the following ways:

• Work With JVM Jobs (WRKJVMJOB), which is new in the 6.1 release, can display the call stacks for the JVM job. Note that WRKJVMJOB is only supported for the IBM Technology for Java JVMs.
• You can use the Retrieve Call Stack API (QWVRCSTK) to retrieve the call stack for the specified thread.
• The Materialize Invocation Stack (MATINVS) MI instruction can be used to materialize the invocation stack of the current thread, or the initial thread of another process.
• When you're using debug by doing a Start Debug (STRDBG), you can use the Display Debug (DSPDBG) command to show the status of debug mode, which will include the call stack.
• You can use the Dump Job command to dump the call stacks for all threads in the job. If you want to dump a job other than your own, use the Start Service Job (STRSRVJOB) command to identify the job (be sure to use End Service Job (ENDSRVJOB) when you done). The following DMPJOB invocation will dump the call stack entries: DMPJOB PGM(*NONE) JOBARA(*NONE) ADROBJ(*NO) JOBTHD(*THDSTK)
• The QTRC APIs allow you to dump out call stack information into a job trace. I'll talk about the QTRC APIs more in a future blog post.

Other tools, such as Job Watcher and Performance Explorer, can include call stack information in their data collections. Having the call stack information as part of these data collections can be extremely valuable in doing diagnostic activities.

Dump a Communications Trace

A recent enhancement to the 6.1 communications trace support allows the collected communications trace data to be dumped to a file in the packet capture (PCAP) format. The Dump Communications Trace (DMPCMNTRC) command was introduced in V5R3. This command lets you dump a communications trace to a stream file. Prior to the 6.1 enhancement, you could then print the dumped trace data with the Print Communications Trace command. Why would you want to dump a communications trace to a file rather than print it? There are a couple of reasons:

• If you're using IPv6 and want to format the trace for the IPv6 headers, you must first dump the trace to a file, then print the trace.
• By dumping the communications trace to a file, you can start another trace on the same communications line, yet keep the prior trace data; when it's been dumped, you can print it multiple times with different options if you want.

IBM recently added support to dump the communications trace data into a file in the PCAP format. Dumping the data in the PCAP format is pretty simple -- you need to create a data area named QCMNTRCPCP in QGPL; the existence of the data area controls whether the communications trace data is dumped in the IBM i internal format or the PCAP format. Dumping the communications trace data in PCAP format lets you use other tools, such as Wireshark or tcpdump, to analyze the trace data.  These tools offer formatting and analysis options that the IBM i Print Communications Trace (PRTCMNTRC) functionality doesn't support. 

You can expect in the next release after 6.1 the DMPCMNTRC command will be enhanced with new command parameters to specify the dump is to a file in the *PCAP format and you will no longer need to create a data area to enable the function.

This function is available with PTF SI33685. The PTF cover letter has the documentation for this enhancement.

 

i Can ... Work With Your JVMs

This week's post was written by Arv Fisher. Arv is the team leader of the Java development team in the IBM i development lab. Thanks, Arv!

Have you ever wanted to check the current garbage collection (GC) heap size, check GC performance or display the parameters the JVM was started with? If you have environments that use Java, these questions as well as many others probably come to mind.

The IBM i 6.1 release has three new commands to assist in answering your JVM environment and performance questions:

• Work With JVM Jobs (WRKJVMJOB)
• Print JVM Job (PRTJVMJOB)
• Generate JVM Dump (GENJVMDMP)

The new commands can be used on any job where an IBM Technology for Java (IT4J) VM is currently running.

By default WRKJVMJOB will list all the currently active JVMs regardless of how they were started. GC information, JVM arguments, environment variables, thread information including Java thread names, the capability to see the Java stack frames, native heap sizes and more are available using WRKJVMJOB. (One thing that's really cool here is the environment variables for the job-until WRKJVMJOB, there was no interface to easily display the environment variables of a job other than your own! So WRKJVMJOB has some interesting function beyond just working with JVMs!)

When JVM performance is being investigated, GC information is a requirement. Most will validly choose not to run with verbose GC enabled all the time, and having to restart the JVM just to enable verbose GC isn't always an option. IBM i always keeps track of the last 300 GC cycles for each IT4J JVM. Just use WRKJVMJOB to print the GC Cycle Table to a spool file. The GC information in the spool file is in a somewhat readable form or IBM Support Assistant (ISA) Version 4 can access IBM Monitoring and Diagnostic Tools for Java -- Garbage Collection and Memory Visualizer (GCMV) -- to analyze the GC information. If you have your own favorite tool that analyzes the traditional verbose GC information you can specify -verbosegc on the Java command line or just use WRKJVMJOB to enable or disable verbose GC at any time. On IBM i, we allow you to collect GC information even when the JVM was not started with verbose GC enabled.

Interested in performing an analysis of the Java objects that exist in the GC heap? Use WRKJVMJOB to simplify the task. To perform an in-depth heap analysis, you need to generate a heap dump. In the past on IBM i and on other platforms, you had to start the JVM with some special Xdump options, find the process identifier of the JVM and then issue a command that looked like "kill -QUIT 5342". I know you were thinking the first time you issued the command and probably still do today, "What were they thinking?". On IBM i, we agreed and thought we can help. Without starting up the JVM with any special options, use WRKJVMJOB to request any IT4J JVM to generate a heap dump. We also have the capability to generate a Java or System dump if you run into a situation where it's necessary. Once you have your heap dump, you can use ISA Version 4 to get access to the tool Memory Dump Diagnostic for Java (MDD4J) to analyze the GC heap. There are times when the special Xdump options may be necessary. They still can be specified on the Java command line as you would have done in the past. We have also provided an option in WRKJVMJOB on the Work With JVM Jobs panel (12=Dump) to generate a dump using the options specified on the Java command line.

WRKJOB also has a new option, 45, to invoke WRKJVMJOB if that job has an active JVM. Experiment with WRKJVMJOB; you'll probably find information that can be helpful and is readily available. Be sure to load the latest Java Group PTF to get the most current version of the commands.

The new commands don't support the Classic JVM. IT4J will be the only JVM supported in the follow-on release to 6.1. See the Upgrade Planning Statement for the Java Developer Kit for additional information. 

The complete list of options for WRKJVMJOB is available at the Information Center.

PRTJVMJOB allows you to target a specific JVM and lets you directly print the information you can display under WRKJVMJOB. The complete list of options for PRTJVMJOB is available at the Information Center.

GENJVMDMP allows you to target a specific JVM and request it to generate a Java, System or Heap dump. The dump options are also available under WRKJVMJOB. The complete list of GENJVMDMP options is available at the Info Center.

Additional information on ISA version 4 can be found at the IBM Education Assistant Web site.