IBM Improves the Handling of the Inactivity Timeout

The inactive job time-out (QINACTITV) system value defines the minimum amount of time that an interactive job can be inactive before an action is taken. The action that is taken is defined by the inactive job message queue (QINACTMSGQ) system value. A related system value, time interval before disconnected job ends (QDSCJOBITV), defines how long a job can be disconnected before the job is ended.

The handling of inactive job time out (QINACTITV) is being changed with a 7.1 PTF (the other system values are unaffected) to address two long-standing problems.

1. Accuracy. Prior to this change, a job can be inactive for up to twice the specified interval before an action is taken. For inactivity intervals measured in hours, this level of uncertainty is simply too high.
   
   
2. Performance. A large number of jobs can be ended at once when the subsystems check for inactivity. This can affect overall system performance at the time the jobs are being ended.

The change is available only for the 7.1 release with PTF SI46398. This is a delayed PTF, so it requires an IPL to put in on or to take it off.

First of us, let's review this change as it pertains to accuracy.

Prior to this change, the subsystem would wait for the QINACTITV number of minutes between the times it checked for inactive interactive jobs. Jobs can be inactive for nearly two intervals before the inactivity is detected; if a job runs just after the system checks for inactivity, at the next check it will not be inactive for the full interval, so the timeout will not occur until the following interval. For example, if you have QINACTITV set to 60 minutes, jobs may need to be inactive for nearly 120 minutes for the inactivity to be detected.

The PTF improves the timeliness in which inactivity is identified.

The QINACTITV function is still enforced by the subsystem code periodically checking for inactivity, but with the change, the subsystem tries to check more often to improve the accuracy. In most cases, the subsystem will now check for inactivity every 10 minutes and will keep track of how long a job has been inactive. If the QINACTITV system value is less than 15 minutes, the subsystem will continue to use the value specified by QINACTITV. (Checking for inactivity every 10 minutes only helps for the cases where QINACTITV is greater than or equal to 15 minutes).

A job can still be inactive for longer than the QINACTITV system value. For any given QINACTITV, you can work out an exact value for how long a job can be inactive before an action is taken. The job becomes inactive sometime during the 10 minutes between checks. So, in the simple case where the QINACTITV value is evenly divisible by 10, a job can be inactive for up to 10 minutes plus the QINACTITV value. But the QINACTITV value does not have to be evenly divisible by 10 and the subsystem cannot take action until the next time it checks for inactivity. For an QINACTITV value such as 101, a job can be inactive for at most the QINACTITV time plus 19 minutes.

Consider an example where QINACTITV is set to 90 minutes. With the old code, the subsystem would take the action defined by the QINACTMSGQ system value if a job was inactive anywhere from 90 minutes to 180 minutes. With the new code, the subsystem will take action if a job has been inactive anywhere from 90 minutes to 100 minutes.

Consider another example. In this case QINACTITV is set to 45 minutes. With the old code, the subsystem would take action if a job was inactive anywhere from 45 minutes to 90 minutes. With the new code, the subsystem will take action if a job has been inactive anywhere from 50 minutes to 60 minutes.

A job will be inactive for at least the QINACTITV value before action is taken. But sometimes you might be interested in "at most" rather than "at least". If you have a requirement that jobs must not be left inactive for longer than some specified value, you need to set the QINACTITV value to something less than the limit that you want to enforce. With the old code, the QINACTITV value had to be half of the limit that you wanted to enforce. With the new code, the QINACTITV value can be much closer to the limit that you want to enforce.

If system value QINACTMSGQ is set to the name of a message queue and the job identified by the CPI1126 message remains inactive and is not ended or disconnected, another CPI1126 is sent the next time the subsystem checks for inactivity. This behavior is unchanged, but the subsystem may be checking for inactivity much more often. This could affect the logic of whatever program is handling messages sent to that message queue. The program will have a lot less time before it gets another CPI1126 message for a given job.

It can still be useful to set the QINACTMSGQ (Inactive job message queue) system value to the name of a message queue so that you can control which jobs get ended or disconnected and how many jobs get processed at a time. Activity can be uneven and can be influenced by external factors such as lunchtime. A significant number of jobs can become inactive within the same 10-minute interval. This is less of a problem with a 10-minute interval than it is with an interval measured in hours. Also, due to other changes in the code, this is less of a problem when the interactive work is spread across multiple subsystems.

Now on to the performance improvements provided by this PTF.

Subsystems that are started at the same time will no longer check for inactivity at the same time. When the subsystems start or when the QINACTITV system value is changed, the subsystems now use a small but variable delay when setting up the checking for inactivity. This helps reduce the number of jobs that are disconnected or ended at a single time. With the old code, you could introduce delays between starting subsystems and these delays would translate more or less directly into delays between the times the different subsystems would check for inactivity. With the new code, every subsystem has to check every 10 minutes, but the system takes care of spreading out the work. Now, the only reason to have delays between starting subsystems would be to manage the amount of work done when starting the subsystems.

When a subsystem sets up its checking for inactivity, it has no idea how many other subsystems will need to do this same type of work. In addition to checking for inactive interactive jobs, subsystems also check the number of prestart jobs that are not being used. PTF SI46398 spreads out the work for ending of inactive interactive jobs and for ending of unused prestart jobs.

Because it is still the subsystem that checks for inactivity, much remains the same. But the system now reacts to inactivity sooner and spreads out the work better. This makes the enforcement of QINACTITV more predictable and reduces the impact on overall system performance.

 

I'd like to thank Dan Tarara for this blog; Dan is a member of the IBM i work management team. Thanks, Dan!

 

Lesser-Known Features of Work With System Activity

Hopefully most of you now know that in the 6.1 release, IBM moved the Work with System Activity (WRKSYSACT) command from the Performance Tools licensed program product into the base operating system. WRKSYSACT has been around for a long time, but it has a few features that you may not be aware of.

Work with System Activity is a great tool for getting a basic understanding of what is happening on your partition in terms of performance. WRKSYSACT will show you jobs, threads, and tasks that are using system resources – CPU utilization, synchronous and asynchronous I/O, storage allocation information, etc. WRKSYSACT is the only green-screen command interface in the operating system where you will find information about tasks. Other command interfaces – such as WRKACTJOB – only show information about jobs and threads.

It would take far too much space to write about all of the things you can see through WRKSYSACT; rather, there are a few features that I want to discuss in more detail.

 

WRKSYSACT in Batch Mode

First of all, did you know that you can run WRKSYSACT in a batch monitoring mode and write the performance data to a file? You enable the batch mode with the OUTPUT parameter; by default this is * and the information is presented on your display. But there are two additional options, *FILE and *BOTH, which will write the performance data to a database file (QAITMON in QPFRDATA by default). You can specify the sample intervals with the INTERVAL parameter (which also controls the auto-refresh intervals as well) and how many intervals are collected (NBRITV parameter). When you take the option to write the data to a file, a job named WRKSYSACT is submitted to QBATCH to collect the performance data.

The Print Activity Report (PRTACTRPT) command can then be used to print out a report from the sample data collected by WRKSYSACT. The PRTACTRPT command can generate a summary report as well as a detailed report and has options to allow you to specify a timeframe of interest, specific jobs you may want in the report, as well as what criteria the report should use in ordering.

 

Wait Information

Wait information will tell you what a particular job, thread or task is waiting on when it is waiting to do more work. I wrote about wait accounting in “i Can Tell You Why You're Waiting” some time ago. You can also see wait information using WRKSYSACT; you simply need to determine which job/thread/task you want to look at, and use option 6, Wait detail, for that job or task. Option 6 will bring up a display that will list the wait categories for that job/thread/task. Be aware that this is a high-level summary of the kinds of wait for that job or task. However, it make give you some clues about what job or task you need more detailed information for; you can then use the Performance Data Investigator with Collection Services or Job Watcher data to view detailed wait information.

 

Storage Allocated/Deallocated

WRKSYSACT will also show the storage allocated or deallocated by a job/thread/task by pressing the F11 key to get to View 4. I wrote about “Understanding Disk Space Usage” and in that blog reviewed WRKSYSACT and the storage allocated/deallocated information.

 

Scaled CPU

Scaled CPU time is a metric that can be used to determine if energy management features, such as power capping or power savings, are in effect on your IBM i partition. These energy management capabilities can be turned on through Active Energy Manager, which is a plug-in to IBM Systems Director. When you use these energy management features, the system's processor may run at lower or higher frequencies, which can affect your partition's performance. I wrote about this topic at length in the blog “i Can Understand Scaled CPU Time.”

On the WRKSYSACT display, the value “Average CPU rate” will be displayed if you are using energy management capabilities. This rate is an easy way to determine if your processor has been slowed down due to power capping or power savings.

One final note – one thing that has not changed with WRKSYSACT is the limit of one user at any given time. WRKSYSACT collects a lot of information from critical interfaces in the operating system and the limit of one user ensures limited contention on those critical interfaces.

 

 

 

 

Instrument Your Applications for IBM i Job Trace

Last week I wrote about the IBM i Job Trace capabilities.In that article, I briefly mentioned that you can instrument your own application so you can collect your own trace data as part of a job trace.

The secret to this support is something called the “QTRC APIs,” which were introduced with the 6.1 release. These APIs are documented under the “Problem Management” category of APIs in the IBM i Information Center. The QTRC API documentation section has much more information than just the programming interfaces; it discusses how you use these APIs and covers the related information you need to know to fully utilize these programming interfaces.

An important concept in how the QTRC APIs work is something called the “component.” The component is a parameter you specify on each of the QTRC APIs to identify the trace collection, and the component is the parameter on the STRTRC command that will turn trace on or off for your component. Below is the documentation taken from the Information Center on a QTRC Component:

Component

A 10-byte character string which names the application component. Each Trace Point has an associated component that is specified on the QTRC Trace APIs. The component name is also shown in the trace output for each Trace Record that is written. A component is defined to the QTRC Trace environment when specified on the Start Trace (STRTRC) CL command. Component names should only contain characters in the invariant character set. Avoid using lowercase a through z. Application component names should contain an abbreviated company name along with the application name. Component names starting with an asterisk (*) are reserved for the operating system. Component names starting with Q are reserved for IBM licensed programs.

There are three basic steps to collect diagnostic data from your application as part of a job trace:

1. Instrument your application to collect the desired diagnostic information.

This is the step that will require the most thought. What kind of data will you need for diagnostic purposes? What are the performance implications of collecting additional data? Do you have different types of data you would need for different kinds of problems?

You use the QTRC APIs to collect the data you need.

• Use the Check Trace Level API to determine the trace level that has been set for your component trace. This check will determine whether trace has been turned on for your component; if the required trace is not active, you can easily bypass the data collection logic in your application and avoid adding overhead to your application when trace has not been turned on.
• Based upon the trace level, you then write trace records; you can collect text data, hexadecimal data, and the call stack. The QTRC API documentation in the IBM i Information Center describes how to use these APIs.

2. Collect your trace data using the STRTRC command.

The magic to using the STRTRC command to collect your own trace data is on the TRCTYPE parameter. While IBM has defined special values to start trace collection for components within the operating system, the TRCTYPE keyword is not restricted to just these special values; you can specify your own component name on the TRCTYPE parameter.

The operating system components that you can trace with the TRCTYPE parameter are using the QTRC APIs as well – the QTRC APIs allow you to instrument your applications in the exact same way that IBM developers have instrumented the operating system.

3. Interpret the collected data by printing out the trace with the PRTTRC command.

There will be trace information included automatically by the system in addition to the component-specific data you add by using the APIs.

Traces can be large so you want to ensure that you clean up after yourself. Once you have completed diagnostics with the trace data, be sure to delete the trace data with the DLTTRC command. You should also delete any spooled trace files when you are done with them as well.

Adding diagnostic information to an application can be time-consuming, but the improved diagnostic capabilities provided by having trace data can ultimately save you a lot of time when problems arise. IBM i has significant tracing capabilities built into the operating system and you can be sure that every developer thinks about how they are going to debug problems that may arise. And with the QTRC APIs, you can do the same thing with your application.

 

 

 

Working With Your Spooled Output

In recent blog articles I wrote about Job Log Pending and the Job Log Output attribute that can be used to prevent the generation of spooled job logs. In addition, in a prior blog, I also wrote how to Detach Spooled Files from a job.

These all can be very helpful features, but one common concern I hear when I discuss these configuration options with clients is how to find their spooled files and job logs. Taking advantage of these features means you will likely need to change your approach to finding spooled files, but the capability to do so has been built into the operating system.

Everyone who has been using IBM i's character-based user interface (the “green screen”) is familiar with the Work with Spooled Files (WRKSPLF) command. This command has been part of the operating system “forever.” However, when a function is part of the base operating system, we don't always realize enhancements have been added because IBM always does it in a way that is upwardly compatible. That is, you can keep on using WRKSPLF just as you always have and not even realize that new features are available. And IBM does not change-flag enhancements to CL commands in the Information Center, which makes it all the harder to know what has changed. And the IBMers that talk about the latest and greatest additions to the operating system generally don't get too excited about incremental enhancements to that “old stuff”.... So while WRKSPLF command has been around “forever” and although the major enhancements were made to it in 5.4 (a “long” time ago), I've discovered that many people are still not aware of these changes.

As a quick exercise, let's look at the command parameters between V5R2 and 7.1.

Here is the Work with Spooled Files Command syntax diagram from the V5R2 Information Center (remember those?!)

>>-WRKSPLF------------------------------------------------------>

>--+-----------------------------------------------------------------+-->

   |          .-*CURRENT--.  .-*ALL--------.  .-*ALL------.  .-*ALL------.  .-*ALL-------.    |

   '-SELECT(--+-*ALL------+--+-*OUTQ-------+--+-*STD------+--+-user-data-+--+-ASP-number-+--)-'

              '-user-name-'  '-device-name-'  '-form-type-'

 

                               (1)

>--+-------------------------+---------------------------------->

   |          .-*INLENV-.    |

   '-DSPFMT(--+-*S36FMT-+--)-'

              '-*NORMAL-'

 

>--+---------------------------+--+------------------------+---><

   |          .-*PRV------.    |  |          .-*------.    |

   '-ASTLVL(--+-*USRPRF---+--)-'  '-OUTPUT(--+-*PRINT-+--)-'

              +-*BASIC----+

              '-*INTERMED-'

Below are the Work with Spooled Files Command parameters from the 7.1 Information Center, but I've changed the Notes column to identify changes to the WRKSPLF command. You'll observe that major changes were done in the 5.4 release.

 

 

Keyword	Description	Choices	Notes
SELECT	Select files for	Element list
	Element 1: User	Generic name, name, *CURRENT, *ALL	Generic name support added in 5.4
	Element 2: Print device	Name, *ALL, *OUTQ
	Element 3: Form type	Character value, *ALL, *STD
	Element 4: User data	Character value, *ALL	Generic support added in 5.4
	Element 5: ASP	1-32, *ALL, *ASPDEV	*ASPDEV added, extended to 32 in V5R3 for IASP support
	Element 6: Spooled file	Generic name, name, *ALL	Added in 5.4
JOB	Job name	Single values: *ALL Other values: Qualified job name	Added in 5.4
	Qualifier 1: Job name	Generic name, name
	Qualifier 2: User	Generic name, name, *ALL
	Qualifier 3: Number	000000-999999, *ALL
PERIOD	Time period	Element list	Added in 5.4
	Element 1: Start time and date	Element list
	Element 1: Beginning time	Time, *AVAIL
	Element 2: Beginning date	Date, *BEGIN
	Element 2: End time and date	Element list
	Element 1: Ending time	Time, *AVAIL
	Element 2: Ending date	Date, *END
ASPDEV	ASP device	Name, *, *SYSBAS, *CURASPGRP	Added in V5R3 for IASP Support
OUTPUT	Output	*, *PRINT
DSPFMT	Display format	*INLENV, *S36FMT, *NORMAL
ASTLVL	Assistance level	*PRV, *USRPRF, *BASIC, *INTERMED

How can these additional parameters help you?

You can use the WRKSPLF command to list spooled files by their file type – e.g., on the command prompt if you type QPJOB* for the Spooled File element on the SELECT parameter, you will display all the spooled files that begin with QPJOB* for your user profile (i.e., all your spooled job logs).

You can use the WRKSPLF command with the JOB parameter to work with the spooled files for a specific job or a generic job specification. For example, if you are detaching spooled files from their jobs, you can use the JOB parameter on the WRKSPLF command to locate the spooled files, even through the job is no longer in the system. Note the JOB parameter requires the USER element of the SELECT parameter to be *ALL.

You can use the WRKSPLF command with the PERIOD parameter to find spooled files from a specific timeframe. You can combine the JOB and PERIOD parameters to further refine the search criteria on which spooled files you are looking for.

And remember, too, that the Work with Job Logs (WRKJOBLOG) command was also added in the 5.4 release to make finding and working with spooled and pending job logs easier.

 

 

 

Job Log Output

Last week I wrote about Job Log Pending. With job log pending, you do not have to write job logs to spooled files.

At the end of that blog, I mentioned how you can use the Log Output (QLOGOUTPUT) system value or job attribute to take advantage of job log pending. This week, I'll explain the values available for “Log output” and give some general recommendations.

There are three options for the log output system value: 

• *JOBEND – the job log is written to a spooled file when the job is ended. This is the default value and is compatible with prior releases and how job logs were generated.
  
  
• *JOBLOGSVR – when the job ends, the job log remains in a pending status until the job log is written to a spooled file by the job log server.
  
  
• *PND – the job log is not written to a spooled file, but remains in a pending status.

While the shipped value for log output is *JOBEND, IBM recommends that you use the job log server. Using the job log server has a major advantage - the generation of the spooled job logs is handled in a more controlled manner. Remember the scenarios I described in last week's blog where you could potentially have hundreds or thousands of jobs writing job logs at the same time? That is a significant drain on system resources. By having the job log server generate the spooled job logs, the system controls the amount of work that is done at one point in time; if there are a lot of job logs to be written, the job logs will remain in a pending state until the job log server can write the job log to the spooled file.

But do you really need all those job logs written to spooled files? If you don't, you should think about using the pending attribute. Job logs that are in pending status will never have their job log written to a spooled file unless you take explicit action to do so. The system's automatic cleanup function will removing pending job logs in the same manner as it removes spooled job logs. If you need to display or print a pending job log, you can use one of the following methods: 

• Use the Display Job Log (DSPJOBLOG) command and direct the output as required.
• Use the WRKJOB or WRKUSRJOB command on ended jobs with pending job logs, similar to how you would work with an active job.
• Use the Work with Job Logs (WRKJOBLOG) command.
• Use the Change Job (CHGJOB) command to change the Log Output value from *PND to *JOBEND. This will write the job log to a spooled file and move the job to output queue status.

 

Job Log Pending

Job log pending has long been a possible state for a job log. When a job has a pending job log, the job has ended but the job log has not yet been written to an spooled file. In older releases of IBM i, this could occur when you powered down the system – the job logs would be saved in a pending state and the job logs would be written to spool files when the system was powered back up.

Every job on the system has a set of job structures, as I described in the IBM i Job Structures blog earlier this year. Job log messages are stored in an internal structure that is called the “job message queue.” The “traditional” behavior is for job log messages to be written a spooled file when the job ends.

However, think about certain scenarios: You end a subsystem that ends many jobs. You have a network failure that causes all Telnet sessions to be terminated. You have a runaway job in a loop that creates an unexpected number of jobs that do a small amount of work and then end. Or other situations where a large number of jobs are ended at about the same time.

In each of these scenarios, you can potentially have hundreds or even thousands of jobs that will all end at about the same time. If all of these jobs that are ending are also writing their job logs to spooled files, you can imagine the additional workload that would put on the system - writing job logs is a fairly expensive operation. Do you need to have all those job logs written? Do you have a need to save those job logs at all?

In the 5.4 release, IBM made job log pending a much more valuable and available option to use for job logs.

Rather than writing job logs to spooled files, job log pending allows you to keep the job logs in the internal form that is used when a job is active. No additional system resources are required to keep a job log in pending form (other than the job structure used to store information about the ended job).

Job log pending is controlled by the Job log output system value (QLOGOUTPUT) or the Job log output attribute on job descriptions.

There is much more to be written about job log pending, which I will do in subsequent blogs.

 

 

 

Clone an Integrated Windows Server

In a Windows server environment, it's normal to have a need to quickly create another Windows server to be used for a new application or even as a backup system to test the latest service pack without interfering with your production server. The problem though is creating multiple servers from scratch gets tedious when needing to install additional device drivers, service packs and common applications after the operating system is installed. But with a recent new-function PTF, support for cloning integrated Windows servers is possible.

An easy-to-use GUI Wizard duplicates an existing iSCSI based Windows 2003 or Windows 2008 server by copying the network server description (NWSD), network server storage (NWSSTG), virtual Ethernet line descriptions, and then makes them ready for that first activation. This support was added to IBM i 7.1 in December 2010, and IBM i 6.1 in May 2011.

Although it's possible to duplicate a single server for test purposes, the primary reason cloning support was added was for rapid server deployment where multiple blades or System x servers are designated for Windows server installations. The first task in creating servers based on an existing server is to ensure your base server is ready to be cloned. This means your server is fully installed with a supported Windows server operating system, latest service packs and any applications that are needed on the clone servers. Once the base server is ready to be cloned, a Windows tool named “SYSPREP” is used to remove server specific information such as log files and license key information and prepares it for a first-time run welcome experience. Information on how to access SYSPREP on Windows 2003 or Windows 2008 is available from the following link: http://publib.boulder.ibm.com/infocenter/toolsctr/v1r0/index.jsp?topic=%2Fdpsccm%2Fdpsccm_t_running_sysprep.html

The SYSPREP command suggested for the integrated server environment should be run as follows to prepare the cloned server to run the mini out of box experience when first started.

• Windows 2003:  SYSPREP -reseal -mini
• Windows 2008:  SYSPREP /generalize /oobe

After SYSPREP has been run, shut down the server and start making as many copies as needed. To start the server cloning wizard, use the Web GUI, select Create Server task and select the Base on an existing object option and specify the NWSD name of the server to be cloned as well as the name of the new server. Continue through the wizard to copy the storage, specify a compatible remote system, and then click Finish on the Summary page. The progress bar will show that status of the creation process, which will take some time duplicating large storage spaces.

The clone server is created with the same properties as the base server with one exception. The virtual Ethernet point-to-point connection for the clone server will have a different TCP/IP interface on IBM i and will need to be manually set on Windows once the server is activated to be in the same subnet.

For detailed instructions on the entire cloning process and using the integrated server cloning wizard, see the section titled “Server cloning road map and checklist” in the IBM i iSCSI Solutions Guide. 

The required PTFs for this support are as follows.

• IBM I 6.1 PTF SI43599 and WebNav GUI service pack PTFs (WSP8)
  or IBM I integration with BladeCenter and System x Group PTF SF99357 Level 20.
• IBM I 7.1 PTFs SI41049, SI41817 and WebNav GUI service pack PTFs (WSP3)
  or IBM I integration with BladeCenter and System x Group PTF SF99369 Level 4.

I'd like to thank Thom Haze from the Integrated Server Development team for writing this blog article.

 

 

 

IBM Systems Director Navigator for i Work Management Integration with Performance Tasks

In last week's blog on the features of the green screen “Work with Active Jobs” interface, there was a comment regarding the nice features of the graphical tools (System i Navigator or IBM Systems Director Navigator). I had written a prior blog on some of the “Cool Things with the Work Management GUI” that covered some additional function within the work management GUI. In that blog I wrote briefly about the enhanced integration between the Active Jobs GUI and the Performance GUI. 

I'd like to expand upon that...

Active Jobs (or WRKACTJOB on the green screen) is often the first place you should look if you want to have some basic understanding of the system's performance characteristics and how the work is running. If you discover some job that appears to be using a lot of system resources, whether CPU or I/O, you may want to dig deeper to understand more about what’s going on. 

In the 7.1 release, there’s much greater integration between the Active Jobs tasks and the Performance tasks that allows you to go from working with active jobs directly to analyzing performance data.

For my examples, you’ll notice I’ve customized the columns in my Active Jobs view–I added the metric “Disk I/O Count,” sorted on the CPU % column, and reordered the columns so that “CPU %” and “Disk I/O Count” were easy to see and include in my screen captures. This type of flexibility with the GUI is great to customize what you need to see.

I can see from Active Jobs that the job QZRCSRVS with current user dmmay (Hey! That's me!) had been using CPU, but also that it had been doing disk I/O. We can use this job as my example job for showing how you can get from an active job to the performance data for it. By clicking on the little arrow to the right of the job name, you bring up a pop-up menu of the options to work with that job; the performance tasks are nested under the performance option, as my screen capture (below) shows.

The performance options to work with Collection Services data are only available when Collection Services is active, which it should be as that’s the default behavior beginning in 6.1. When you select these options to work with performance data from the Active Jobs interface, the most recent Collection Services collection will be used.

If you select the option to Investigate Job Wait Data, you’ll go from the Active Jobs interface to a new tab in your Navigator session to display the Collection Services job wait data for the job. You’ll see something similar to the chart below.

But, in my example, the timeframe is too large and I want to focus in on the data at the current time.  I can do that by zooming in on the chart to display the data more closely.  (At some point, I'll also write more on how to use the Performance Data Investigator).

By zooming in on the performance data on the very right-hand side of the graph, I can see the Collection Services data in much more detail, and I discover that my job has been page faulting, see figure below.

 

With Collection Services data, you can't get more detail about why the job was page faulting. 

However, going back to the Active Jobs interface makes it easy for you to get the detailed data you need to investigate the problem further.

If you have the IBM Performance Tools licensed program product, option 3 (Job Watcher) installed, you will have the ability to start Job Watcher from the Active Jobs interface, again, under the Performance options for that job.

This will launch the Job Watcher wizard to lead you through starting a Job Watcher collection. Once you have the more detailed performance data you can get with Job Watcher, you can use the performance tasks to further investigate the data. Note, however, that you’ll need to reproduce the performance problem while Job Watcher is active in order to have the desired detailed data for analysis.

The steps I just went through were how you can get to performance data for one job by starting from Active Jobs.

You can also look at system wide performance data by starting with Active Jobs.  To do this, you don’t select any particular job to work with, but rather use the action pull-down to select the option to Investigate Jobs Wait Data (you can also Investigate Jobs Statistics Data as well). See figure below.

Displaying the Jobs Wait Data will display the Collection Services job wait data for all jobs, as the following screen capture shows.

 The Job(s) Wait Data show the details of what the job had been waiting on. Refer back to the blog on “i Can Tell You Why You're Waiting” for more information on waits.

The option to Investigate Jobs Statistics Data will present Collection Services jobs statistical data in table form, where you can search and sort, customize columns, etc., to work with the performance data in table form. By default, all columns are displayed in the table, and you must scroll right or left to display them. Just like any other table view in IBM Systems Director Navigator, you can customize which columns you want to see, the order in which the columns are displayed, as well as the sort order you want to use. By looking at Jobs Statistics Data, you’ll discover all sorts of job metrics in Collection Services that may be interesting for analysis purposes.

Work With Active Jobs

This week's blog was written by Mike Russell, who is a member of the IBM i work management team. Everyone knows about the Work with Active Jobs command, but Mike shares some of the lesser known features as well. Thanks, Mike!

This article describes the Work with Active Jobs command. Work with Active Jobs helps you see and manage what is running on your IBM i system. It shows a list of all the active jobs with their current status and basic performance information.

Basic Features

The job type, current user, priority, memory pool, status and number of threads can all be shown by pressing F11. Basic performance information shows the job’s processor usage, paging and interactive response time. Options can be entered to control the job or find more detailed information about a single job. F23 can be used to see all the options available and F24 to see all the function keys available.

For information that can be different for each thread in a job, the initial thread’s information is shown. The status shows whether the job is currently running or not. The status values ending in W show the job is waiting. The same value ending in A means it is waiting in the activity level.

Elapsed interval

Work with Active Jobs gathers current information for each active job when the command is run and each time the screen is refreshed. The current performance information is compared to information collected at an earlier start time. The difference between the current time and the start time is called the elapsed interval. The elapsed interval is used to calculate the CPU percent, number of interactions, average response time and auxiliary I/O operations for each job. The elapsed interval is shown at the top of the screen along with the percentage of CPU used by the system and number of active jobs. When the WRKACTJOB command is used for the first time in a job, the elapsed interval is zero and no performance data is shown. This is the start time of the elapsed interval. The F5=Refresh key causes the end time of the elapsed interval to be set to the current time, the start time remains the same. The F10=Restart key causes the end time of the elapsed interval to be set to the current time, with the start time set to the previous end time. In other words, F10 shows you what has happened since the last time you looked. F5 extends the elapsed interval. F13=Reset statistics can be used to set the elapsed time to zero and reset the display.

If you are looking for performance problems, you want to use F10 occasionally. With a long elapsed time the performance information becomes watered down, making it difficult to see what's currently happening.

Other Features

Sorting and limiting

It can be difficult to find what you are looking for in a long list of jobs. There are features in WRKACTJOB to help you find, sort and limit the number of jobs shown. F7=Find displays a pop-up to allow you to enter a search string and what column to search. F8=Repeat find will find the next instance. F18=Bottom and F17=Top will get you to the bottom or top of the list without scrolling.

By default the jobs are grouped under the subsystem they are running in. You can sort the jobs by the values in any column on the display by moving the cursor into the column and pressing F16=Resequence. The SEQ (sequence) parameter on the command can also be used to sort the jobs. And the F20=Subset pop-up also lets you change how the jobs are ordered. Sometimes sorting the jobs into a different order can make it easier to see which jobs are using the processor, work with all jobs in MSGW status, or monitor the paging of jobs running in a memory pool.

By default all active jobs are shown except those that are normally expected to be idle. Idle jobs include the suspended system request and group jobs, disconnected jobs, and prestart jobs that are waiting for a request. You can show the idle jobs by pressing F14=Include. There are several parameters on the command to let you work with a smaller list of jobs. The CPUPCTLMT and RSPLMT parameters let you only see jobs that are using the processor or having long response times. The SBS parameter lets you see only jobs in subsystems you are interested in. The JOB parameter lets you see only jobs matching a generic job name or system jobs or subsystems. The F20=Subset pop up lets you change which jobs are included on the display without exiting the display and entering the command again.

Option 9=Exclude can be used to remove jobs from the display that you're not interested in. This allows you to get the jobs you are interested in on the same screen so you don’t have to roll back and forth to compare their performance information. The jobs remain excluded until you use F13 to reset the display. F21=Nondisplay instructions/keys can be used to get more jobs shown at one time. The option and function key descriptions are removed. F21 again will change back to display the descriptions.

Automatic refresh

WRKACTJOB can be left running to monitor the system activity by pressing F19=Start automatic refresh. The screen will be refreshed every 5 minutes. You can change how often the screen is refreshed with the INTERVAL parameter. In release 7.1, the automatic refresh function was changed to only show activity for the latest interval. In previous releases, the elapsed time increased each interval to show activity for the entire period since automatic refresh was started, making the information less and less useful over time.

Setting the Function field

The function field usually shows a high-level command, menu or program used by the job. You can use the Change Current Job QWCCCJOB API to customize the function field in a long-running program to something more meaningful. USR- will precede the specified value on the display.

 

Multi-Threaded Interactive Jobs?

A comment was posted to last week's blog on IBM i Job Structures. It read:  

Dawn:
This job structure article jogged my memory regarding a recent situation which I witnessed with a Series i customer. They have multi-threaded INTERACTIVE jobs running on their system. Can you sometime explain the concept of multi-threaded interactive jobs? I know and understand multi-threaded batch jobs but know nothing about multi-threaded interactive jobs. How are they created? What purpose do they serve? From a performance standpoint, does QPFRADJ adjust activity levels for multi-threads of interactive jobs? Any additional info would be helpful. Thanks!

An IBM i programmer cannot create secondary threads in an interactive job; the allow multiple threads parameter on job descriptions is ignored for interactive jobs and you will receive an error if you attempt to run a program that starts secondary threads when in an interactive job. This is because interactive jobs are not thread safe. One reason interactive jobs are not thread safe is because I/O to display devices is not thread safe. 

So how can a multithread interactive job be created? 

Threads were introduced to the system in V4R2. In V4R4, the threads support was extended to add “system threads.” System threads are threads of execution used for system functions and can be created in any job, including interactive jobs and jobs that have the allow multiple threads job attribute set to *NO. Because system threads can only be used by operating system developers, there are expectations for how those threads are used to ensure thread safety issues do not occur. Java uses system threads and several DB2/SQL functions in the operating system also use system threads for parallel operations.

You'll find no documentation in the IBM i Information Center on system threads. There are bits of documentation, however, that will give you small clues about them. For example, the command help for the allow multiple threads parameter includes the following statement: “Specifies whether or not the job can run with multiple user threads. This attribute does not prevent the operating system from creating system threads in the job.”

Next week I'll write about threads and the performance adjuster.