IBM i 7.2 - Batch Model

In the IBM i 7.2 release, there is a new function called “Batch Model”; the 7.2 announcement stated: “The Batch Model has been added, providing the ability to change environment variables, modeling how those changes would impact the batch window.” Batch Model is a utility that can help you understand your batch workloads and can assist in predicting the performance result of hardware changes (processors or disk) or an increasing workload on your batch window.

You will find Batch Model under Sizing within the Performance tasks in Navigator, as the following image shows. You must have the IBM i Performance Tools licensed program product (5770-PT1, option 1) installed in order for these tasks to be available.

Batch model runs over Collection Services data; you identify the timeframe that represents your batch application when creating the batch model. Once the model is created, you may need to calibrate the model; this is needed if the system didn’t create the model as you require. For example, your batch application may consist of several jobs and you must ensure those jobs and linkages (parent/child relationship) are correct.

Once your batch model has been correctly calibrated, you then use the Change Batch Model function to change various configuration components of your batch application. You can change the system model (processor) configuration, the disk configuration and the workload job characteristics (move, copy, delete jobs from the model. Once these changes are made, you analyze your batch model to get the results. Once the analysis is complete, you can investigate the results with the Performance Data Investigator. The Performance Data Investigator has a few ways to look at your modeled results, comparing the measured workload you started with to the results of the modeled workload.

One example is the resource utilization overview, as the following image shows:

You can use Batch Model on Collection Services data from the 6.1 or 7.1 releases. You do need to have a 7.2 partition to access the Navigator interface for Batch Model, but you can save your Collection Services data from 6.1 or 7.1 onto that 7.2 partition and select that older performance data when creating your batch model. This is very handy for sizing exercises if you are planning a HW or SW upgrade. Note: Support for the POWER8 hardware models is not yet available but will be coming soon in a PTF.

The development team that worked on creating Batch Model has provided an article, “How to use the Batch Model performance tool” on developerWorks, where you can find a more in-depth review of this tool.

Some of you may be familiar a tool called Batch/400. Batch/400 was a utility written many years ago as an as-is tool to help predict the impact of changes on a batch application; it had a command called BCHMDL. While I was writing this article, I thought perhaps I might be able to find some historical references for the Batch/400 utility. It turned out to be a bit difficult to find much documentation about it. An Internet search on “Batch/400” revealed “Batch 400 American Stout brewed by Swamp Head Brewery.” An Internet search on “BCHMDL” told me about the British Columbia Hockey League … with a couple results more along the line of what I was really looking for.

 

Sending a Spooled File via TCP When Working With Spooled Files

You can work with spooled files through a variety of command interfaces: Work with Spooled Files (WRKSPLF), Work with Output Queue (WRKOUTQ), and Work with Job (WKRJOB) option 4 (aka WRKJOB OPTION(*SPLF)). These interfaces have option 1=Send, which can be used to send spooled files to another system; this send option invokes Send Network Spooled File (SNDNETSPLF).

This was a very convenient way to send a spooled file to another system using SNADS when SNA was the networking protocol of choice. SNA, as well as SNADS, is not as widely used today. A few clients still use SNA and SNADS, but most have moved away from SNA to exclusively use TCP/IP and use the Send TCP Spooled File (SNDTCPSPLF) command to send spooled files via TCP/IP.

Last fall, a PTF for 7.1, SI50975, was delivered to provide the capability to modify the send option when working with spooled files. Users can now use SNDTCPSPLF rather than SNDNETSPLF. This function is not available on 6.1.

Note: The 7.1 Print PTF Group (SF99366) does not yet include this PTF, so even if you have installed the latest print group, you’ll still need to download and install SI50975.

To change the behavior to use SNDTCPSPLF rather than SNDNETSPLF, you need to add an environment variable. From the SI50975 PTF cover letter:

To change the "1=Send" option on the WRKSPLF, WRKOUTQ, and WRKJOB
OPTION(*SPLF) panels to use SNDTCPSPLF instead of SNDNETSPLF, add a new environment variable QIBM_SNDSPLF_OPT with a value of 'TCP'. The environment variable name and its value must be uppercase, and may be set at the *SYS level or *JOB, or both. Example:
> ADDENVVAR ENVVAR(QIBM_SNDSPLF_OPT) VALUE(TCP) LEVEL(*SYS)

After you have added the environment variable, when you select option 1 on the WRKSPLF command, you will now see the prompt for SNDTCPSPLF.

If you are using the GUI (either the Navigator for i Web console or the System i Navigator application), this PTF does not matter since the GUI already provides the ability to send spooled files by both TCP/IP and SNA:

IBM provided this PTF as an ease-of-use enhancement for working with spooled files via the command line interface. You can also extend WRKSPLF yourself by using the UIM Spooled File User-defined List Action (QIBM_QSP_SPLF_LSTACT) exit point, which allows you to define additional list actions on the following green-screen interfaces:

• • Work with Printer Output (WRKSPLF ASTLVL(*BASIC))
• • Work with Spooled Files (WRKSPLF ASTLVL(*INTERMED))
• • Work with Job Spooled Files (WRKJOB OPTION(*SPLF))
• • Work with Output Queue (WRKOUTQ output queue name)
• • Work with Spooled File Status (D P or WRKSPLF DSPFMT(*S36FMT))

There is an example program in QUSRTOOL/QATTCL.TSPUIML that uses QIBM_QSP_SPLF_LSTACT to add a 'C' option to the WRKSPLF, WRKOUTQ and WRKJOB panels to do a CPYSPLF, and a 'T' option to do SNDTCPSPLF. The biggest disadvantage of using this exit program support is the additional list actions do not show up in the "1=Send 2=Change 3=Hold 4=Delete..." options list at the top of the screen.

IBM Support also has a useful article, Adding Options to WRKSPLF, WRKOUTQ and WRKJOB to Invoke the CPYSPLF and SNDTCPSPLF Commands, on using the exit point, which is what you will need to do if you want this capability on the 6.1 release.

 

 

IBM Power Systems Webinars

Here’s a great resource that you may not be aware of… the IBM Power Systems technical webinar series. These webinars were started way back in 2011 and have been ongoing since. They are geared to sharing technical information related to Power, PowerVM and a virtualized power environment. 

As IBM i clients start to venture into this virtualized world, there are a lot of things to learn. These webinars are a great resource to assist with that learning process. All of the older presentations are still available, which includes the replay of the webinar and the presentation PDF.

There webinars are generally held monthly, and the upcoming ones for the rest of 2013 are published on the website.

You do need to register to participate in the webinars, but it’s a one-time registration.

So if you are starting to get interested in virtualization topics on Power, take advantage of this treasure trove of information! 

More Changes for the Handling of QINACTITV

A little over a year ago, IBM changed the handling of the inactive job time-out (QINACTITV) system value to improve accuracy and performance. That change is described in the blog article IBM Improves the Handling of the Inactivity Timeout.

IBM has made some additional improvements to the handling of QINACTITV. This time, the changes address two questions:

1. What does it mean for an interactive job to be inactive? 
With the change, a job can use CPU time and still be considered inactive.

2. Which jobs can be disconnected and which jobs must be ended? 
With the change, a job can be disconnected even if a device name was not supplied when the session was created. What this means is that jobs for devices whose names start with QPADEV can be disconnected. This affects the Disconnect Job (DSCJOB) command processing and the inactivity timer processing, but it does not affect the handling of device errors.

The change is available only for the 7.1 release with PTF SI50502. This is a delayed PTF, so it requires an IPL to put it on or to take it off.

First, let's look at what it means for a job to be inactive.

• Is there I/O occurring?
  We first look at the session – the connection between the user and the system. A session is inactive if there is no input or output occurring. A session is active if the system continues to send data to the user even if the user never sends input to the system.
• Who is waiting?
  A session will be inactive while the system is working on a long-running command, because there is no I/O occurring. In order for the job to be considered inactive, the system must be waiting for input from the user. The Work with Active Jobs (WRKACTJOB) command shows a status of DSPW for a job that is waiting for input from a workstation.
• Is the job using CPU?
  Looking at whether the job uses CPU time can be both helpful and harmful. Let’s look at some examples and how the new PTF affects these examples.
  
  The job might do work that comes from sources other than the workstation. For example, the job may have a message queue allocated and a break message handling program defined. In this case, using CPU indicates that the job is doing work and is in some sense active. With the new PTF, running a break message handling program will no longer be a reason to consider the job active. The new code gives less weight to unpredictable, background activities.
  
  The job might do work that is just a part of being there and interacting with other jobs. The job uses CPU time, but there is no reason to consider the job to be active. For example, when some other job looks at the invocation stack of an interactive job, the interactive job does some of that work and is charged with the CPU time it uses. This might occur when you have the inactive job message queue (QINACTMSGQ) system value set to the name of a message queue. The program that handles the CPI1126 message might want to look at an invocation stack to decide if a job should be considered inactive. With the new PTF, looking at an invocation stack will no longer be a reason to consider the target job to be active.
  
  More and more things are being done in the background, often in secondary threads. Using JAVA code in an interactive job is one way to see CPU time used just for being there and interacting with the environment.

There is no perfect way to decide whether a job should be considered active. A program handling CPI1126 messages can decide what to do with a job that the system considers inactive, but it is not notified about a job that the system considers active. The PTF cover letter describes a way to use an environment variable to change how the system treats CPU time for determining inactivity, but it cannot solve all the potential problems. The session will look active when someone sends a message to the workstation message queue, even if the user for that workstation isn't there.

Now on to the question of which jobs can be disconnected.

In order for a Disconnect Job (DSCJOB) to make sense, there has to be the capability to reconnect to the job. In order to reconnect to the job, the same user must sign back on to the same device description.

For the case where DSCJOB is being done in response to a device error, DSCJOB is not allowed for sessions where no specific virtual device is requested and the system selects which device to use. The session is gone and it is very unlikely for that user to get the same device when the user next connects to the system or when the device is next used.

For inactivity, the session is still intact and connected to the same user's workstation. The DSCJOB command is used by programs that handle the CPI1126 message when the QINACTMSGQ system value names a message queue. That means the command should act the same way the inactivity timer code acts. Also, if a user wants to DSCJOB rather than SIGNOFF, this is probably a good thing to allow.

This PTF allows more jobs to be disconnected. If your system is set up with QINACTITV(*NONE) or QINACTMSGQ(*ENDJOB), this PTF will probably have very little effect on the workload. So, for discussion, we'll assume that jobs are being disconnected for inactivity.

There is a cost to start a new job, to end a job, to disconnect a job, or to reconnect to a job. Different applications will have different costs. While the overall amount of work is important, many systems are more strongly affected by how that work is distributed over time. Disconnected jobs continue to hold locks and use system resources. This is also true of inactive jobs.

1. When a user signs on and is reconnected to a disconnected job, the system avoids the cost of ending the old job and creating a new job. When a large number of users reconnect, the work is generally well spread out over time.
2. When a user does not reconnect before the time limit defined by the time interval before disconnected job ends (QDSCJOBITV) system value, the disconnected job is ended. By comparison to QINACTMSGQ(*ENDJOB), the system sees the extra work of a disconnect, but there is no work avoided. All jobs see the same QDSCJOBITV value, so the work is spread out the same way the work for QINACTITV is spread out.
3. When a user does not reconnect and the subsystem is ended before the QDSCJOBITV time limit, the work of ending the disconnected jobs gets done all at once during the ending of the subsystem. This can significantly increase the stress on the system. Jobs that are inactive and not disconnected are also ended during the ending of the subsystem.

The QDSCJOBITV system value should be set high enough to allow users enough time to reconnect to their disconnected jobs, but low enough that the disconnected jobs are likely to be ended before the subsystem ends. If users are not going to reconnect, the QINACTMSGQ should be set to *ENDJOB rather than *DSCJOB.

One of the common problems with QINACTITV occurs when a user returns to a session at the same time the system is checking for inactivity. The system does not know that the user is there until the system sees input on that session and the system only sees input when the user presses enter or a function key.

If QINACTMSGQ is set to *DSCJOB and the user is using a virtual device selected by the system (a QPADEVxxxx device), the job will now be disconnected rather than ended. The user can sign on and continue. The PTF should be very helpful, even though the handling of QINACTITV will never be perfect.

 

I’d like to thank Dan Tarara for writing this blog. Dan is a member of the IBM i Work Management development team. Thanks, Dan!

Starting TCP When the System is IPLed

Years ago, when TCP/IP was starting to become the networking standard of choice, it was not automatically when the system was started. You had to edit the system start-up program to automate starting TCP/IP and related interfaces and servers.

In the V5R1 release, the work management component provided the ability to automatically start TCP/IP when the system was started with the addition of the IP attribute “Start TCP”. The default is *YES, but you can change it to *NO to control the starting of TCP/IP yourself.

When the system automatically starts TCP/IP, it does so after the controlling subsystem has started; thus this support works for coming out of restricted state as it does for a partition IPL. When starting from an IPL or restricted state, work management also starts the QSYSWRK subsystem. QSYSWRK is required for TCP/IP functionality since many of the daemon jobs run in that subsystem; the start TCP/IP processing job (QSTRTCP) also runs in QSYSWRK.

When the STRTCP command is left as shipped by IBM, the start TCP processing will first start the defined IPv4 interfaces, the IPv6 interfaces, and the Point to Point (PPP) profiles. The STRTCP processing submits a job named QTCPSTSVRS which starts the application servers. This job is not synchronized with other STRTCP processing, so it can begin to start servers before the interfaces have all been started. TCP Server start-up processing should handle these dependencies and have retry logic if necessary. You configure TCP application servers to start with TCP/IP by specifying the “Start with TCP/IP is started” or the “Autostart” attribute. The blog I wrote earlier this year, Automatically Starting TCP Servers when TCP/IP is Started, discusses this in more detail.

Another important change occurred in the 6.1 release. Prior to 6.1, when TCP/IP was started, it would start the QTCPIP and QTCPMONITR jobs in the QSYSWRK subsystem and these jobs controlled TCP/IP. However, if these jobs were ended, then TCP/IP would also end. To improve the reliability and availability of TCP/IP, these jobs in QSYSWRK are no longer used and the QTCPCTL and QTCPWRK system jobs are used instead. The QTCPCTL job is responsible for managing requests that are sent to the QTCPWRK job. The QTCPWRK job manages the starting and ending of TCP/IP, and the starting and ending of interfaces and PPP profiles. The QTCPWRK job also acts as a message relay agent for TCP/IP related messages. This allows the TCP/IP protocol stack to remain active when in restricted state. Typically, the restricted state is for tape back-ups that use TCP/IP. Refer to the 6.1 Memo to Users for more information on the change to system jobs for TCP/IP processing.

I have heard about clients having issues reliably stating TCP/IP servers when starting TCP/IP at IPL or when starting from restricted state. This can be due to the dependencies between subsystems and the server jobs. Some server jobs run in subsystems other than QSYSWRK; some servers allow you to customize the subsystem in which they run. If the subsystems required by a TCP server are not started before the server jobs are started, problems can result. If you have problems with TCP/IP servers automatically starting after an IPL or coming out of restricted state, you can customize the system's start-up program (specified by the QSTRUPPGM system value) to control starting the subsystems and server jobs.

IBM has a support knowledge-base article titled “Starting TCP - Using the IPL Attributes Versus the System Startup Program”. If you call IBM Support with a problem related to the STRTCP processing at IPL, the IBM support representative may discuss the information documented in this article.

However, the STRTCP IPL attribute processing has improved over the years and using that IPL attribute is the recommended way to STRTCP. If you have specific errors when using the IPL attribute to start TCP/IP, IBM support will need to know the details (Is TCP not starting as a whole? Is a specific server not starting? etc.). It is possible that more complex environments may need to use a customized start-up program to get all the servers correctly started.

However, if you encounter server start-up problems, or if you can recreate problems that cause you to add DLYJOBs to your start-up program, you should open a PMR so the IBM development team can understand the issue and work on a resolution.

Finally, the IBM i Information Center has a section on TCP/IP Troubleshooting. Within that documentation there is a section on Timing Considerations that reviews the general considerations for setting up your own start-up program to start TCP/IP and TCP Servers.

 

 

 

Multi-Threaded Job Table Cleanup

I'd like to thank Mike Russell for writing most of this blog article. Mike works on the IBM i Work Management team and was responsible for the changes described in this blog. Thanks, Mike!

 

Some time ago, I wrote about the job table and permanent job structures in IBM i Job Structures (January 11, 2011). That article reviewed the job table and also discussed compressing the job table. 

In the IBM i 7.1 release, IBM restructured the processing of the job table cleanup that at happens during an abnormal IPL as well as the processing that is done when job tables are compressed. The job table cleanup IPL step is also known as Work Control Block Table cleanup (C900 2C40) IPL step.

Prior to 7.1, the job table was processed by one thread during the IPL step; when the job table became large, it could take this one thread quite some time to process all the entries in the job table. In the 7.1 release, this single-threaded processing was changed to be multi-threaded. This change decreases the time the system is unavailable after an unexpected termination as well as when the job tables are compressed during an IPL.

This new news is that this function is now available for the 6.1 release with PTF SI48088.

The performance improvement is seen on systems with a large number of entries in the job table. You can use the Display Job Table (DSPJOBTBL) command to view the number of entries on your system. The Change IPL Attributes (CHGIPLA) command allows you to influence how much cleanup is done at IPL with the Check job tables (CHKJOBTBL) and Compress job tables (CPRJOBTBL) parameters. How the system powered down is the other influence on how much cleanup is done at IPL. You can check the history on your system to see if you have had any recent abnormal IPLs with the Display Log command: DSPLOG PERIOD((*AVAIL *BEGIN)) MSGID(CPI0990 CPI091D)

There is also a 7.1 release PTF (SI48454) that includes a minor performance improvement for job table cleanup and a correction for the IPL status estimated time remaining:

 

You should not have to run job table cleanup or compression very often, but it is one of those things that when it is necessary, time is most likely important as well. This was an important fix for reducing IPL time in the 7.1 release and it's now available on 6.1.

Work With Job and Select a Thread …. to Print Thread Stacks

In the 7.1 release, the Work with Job (WRKJOB) command was enhanced; you can determine this by going to the CL Command Finder and take the option to display “All changed commands.” The Information Center will show you a list of changed commands and you can read the documentation on the command. Of course, there's a slight problem in that commands are not change-flagged, so you cannot actually see what changed. You need to compare the command documentation with that of the prior release to figure out exactly what has changed.

By comparing the 6.1 version of WRKJOB with the 7.1 version, you will discover that IBM added a new parameter, Select Thread (SLTTHD) to the command. Upon first glance, it might appear that this allows you to simply work with a specific thread for that job; maybe assuming it's a “fast path” way to do a WRKJOB followed by Option 20 (work with threads).

But that is not the case.

A blog I wrote nearly three years ago, Display Call Stacks, discussed the various ways you can access call stacks, as well as some enhancements that were made to the call stack itself in the 6.1 release.

What do these two things have in common?

The Select Thread parameter provides an interface to print the thread stacks for the initial thread of a process (the default), selected threads in a process, or all threads in a process. Prior to this enhancement, the only way to access all thread stacks within a job was to do it interactively; if you specified OUTPUT(*PRINT), it would just print the call stack of the initial thread. A better way was needed to capture integrated thread stacks (which include the IBM Technology for Java stacks and PASE stacks) for diagnostic purposes, which the addition of the SLTTHD parameter on the WRKJOB command provided.

Here's how you would print the call stacks for all threads within a specified job:

WRKJOB JOB(######/user/jobname) OUTPUT(*PRINT) OPTION(*PGMSTK) SLTTHD(*ALL)

And here's how you could select one or more threads from a list; if you select multiple threads from the list presented, each thread's call stack will be printed in the single spooled file that is generated:

WRKJOB JOB(######/user/jobname) OUTPUT(*PRINT) OPTION(*PGMSTK) SLTTHD(*SELECT)

This enhancement is sneaky is because the new parameter, SLTTHD, was hidden from the WRKJOB command prompt – you need to press F9 in order to see it, and because it requires the specific combination of OUTPUT(*PRINT) and OPTION(*PGMSTK) for the option to be valid.

If you have a need to generate spooled output of thread stacks for a job, this handy command invocation will do it for you.

Job Performance Information

Last week I wrote about the basic performance information that had been added to the CPF1240/CPF1241 messages, and in that blog also mentioned the basic performance information in the CPF1164 message that is sent to the history log.

As reader Tim Hawkins commented:

If you really want granular performance information (without the overhead of the Performance Tools assuming you have it), use the Job Accounting journals, there are several tools available to convert them to DB files. Then query the data using your favorite tool.

Tim, not realizing it, was the “straight-man” for this week's blog; although I didn't state it, when I was writing last week's blog, I was planning to write next about job performance data, the options available, and when you might chose which option.

IBM i has very rich features regarding performance metrics; while the capabilities are very comprehensive, it can sometimes be confusing as to what feature you should use when. Of course, the answer to every question related to performance is “It depends!” I've not written about much of these capabilities since I speak on them frequently, but you can be sure they are on my list of future topics.

This week, I'll review the basic options for job performance data and provide brief guidance for when you might consider each option.

• CPF1164, CPF1240/CPF1241 messages as reviewed last week.
  Active Jobs, Server Jobs, System Status, etc...

The commonly used interfaces used such as work with active jobs provide basic job performance information in an interactive interface (either via the command line interface or the graphical user interface). The GUI also has the ability to look at server jobs as a subset of active jobs. These interfaces are useful for a quick view of performance across a small set of jobs.

There are also API interfaces to retrieve this information programmatically.

• IBM i Job Notification Exit Point
  The Job Notification exit point can be used to log notification messages to data queues when IBM i jobs are placed on a job queue, a job starts, or a job ends.
  With the job notification exit point, when a job ends, you will have the date and time the job entered the system, when the job started, and when the job ended. You can also get the processing time used; no other performance data is available.
  
  The job notification exit point would be useful if all you need is awareness of jobs starting and ending, and possibly the processing time used.

• IBM i Job Accounting
  The When to Use Job Accounting article in the Information Center has a nice summarization of when you'd use job accounting data as opposed to when you would use the data in the CPF1164/CPF1240/CPF1241 messages.
  
  Job accounting data includes all the data in the above-listed messages. Job Accounting data includes additional metrics, such as print and communications information (note the communications information is for SNA communications).
  
  You would use job accounting if print and communications information is important to you or the ability to assign accounting codes is required.
  
  Also, the performance data in the CPF1164 message pre-dates the job accounting information and job accounting would be the preferred way to gather this type of job data today. If IBM were to add additional, basic performance metrics, it would be added to the job accounting data and not to the messages. IBM i developers are aware there are customers still using the old method of getting the basic performance data from the messages in QHST so it will continue to be supported.

• The Job Accounting Journal entry documentation details the information that is collected.
  Collection Services Performance data
  
  Collection Services, which runs 24x7 by default starting with the 6.1 release, has comprehensive job performance data collected. This data is stored in the QAPMJOBMI and QAPMJOBOS database files. In addition to the job performance metrics, Collection Services also collects save/restore metrics for a job in the QAPMJOBSR file. Finally, wait information is also collected by Collection Services and provides the ability to determine Why you are Waiting.
  
  If you need low-overhead performance data collection with detailed performance data, Collection Services provides the best solution; when you run Collection Services, it will collect and save detailed job performance data before you know you need it! You can query the data from the files directly, or you can use the Performance Data Investigator to graphically Investigate Collection Services data.

• PM for Power Systems
  You probably are familiar with PM for Power Systems from the “old” name of PM/400. PM for Power Systems summarizes data from Collection Services and optionally sends that summarized data to IBM. By choosing to use PM for Power Systems, you can get summarized trend analysis information on your system's utilization and performance, plus job information (most frequent, processor time, I/O and print) on the top 10 low priority jobs. Remaining capacity is constantly calculated and the data are readily available to the IBM Workload Estimator for sizing future needs.

• Job Watcher Performance data
  The Job Watcher performance data collector was added to the operating system in the 6.1 release. It provides the ability to get detailed job performance data and collects much of the same job data as Collection Services does, but it also collects additional information, such as call stacks, SQL statements, objects being waited on, and more.
  
  Job Watcher is used for detailed job performance analysis and diagnostics. You can view Job Watcher data with the Performance Data Investigator or the iDoctor performance tool.
  
  

As you can see, there are a number of alternatives available. Which one you chose will depend upon the specific requirements you have for why you need job performance information.

CPF1240 and CPF1241 Messages Contain Summary Performance Data

You probably know that the CPF1164 (Job Ended) message that is sent to the history log contains some basic performance information regarding the job that ended. You can't see the details of this information via the Display Message Description (DSPMSGD) command or via the Display Log (DSPLOG) command, but the data is available in the history log files and you can write application programs to retrieve this data.

The basic performance information on the CPF1164 message includes:

• Date and time the job entered the system
• Date and time the job started
• Total response time (for interactive jobs)
• Total transactions (for interactive jobs)
• Number of synchronous auxiliary I/O operations

The Performance Information and QHST section in the Information Center is where documentation on this can be found.

What you may not be aware of, is in the IBM i 6.1 release, basic performance information was also added to the CPF1240 (Job ended abnormally) and CPF1241 (Job completed normally) messages.

Unlike the CPF1164 message, the performance information on the CPF1240 and CPF1241 messages is part of the message description and you can display the information with the DSPMSGD command. For example, a DSPMSGD CPF1240 will show the following information at the end of the technical description information:

   -- Entered system date and time:      &9

   -- Started date and time:             &10

   -- Job type:                          &16

   -- Job subtype:                       &17

   -- CPU seconds used:                  &11

   -- Synchronous auxiliary I/O:         &12

   -- Asynchronous auxiliary I/O:        &13

   -- Ending code:                       &14

   -- Secondary ending code:             &15

And by looking at the field data of the message, you can find the details of the data type and length for each of the replacement variables.

There are a few minor differences from the data that is included with the CPF1164 command.

• The total response time and total transactions fields are not part of the information on the CPF1240 and CPF1241 messages; these messages are only sent for batch jobs and total response time and total transactions are only applicable to interactive jobs.
• Asynchronous auxiliary I/O counts are included on the CPF1240 and CPF1241 messages.

Work With Traces

In the 5.4 release, IBM introduced the Work with Traces (WRKTRC) command. This is a handy little utility that allows you to review all the traces that may be active on a partition, as well as a centralized interface for managing traces.

Why is the WRKTRC a convenient tool? IBM i has very strong diagnostic capabilities and one of the reasons is due to the wide variety of diagnostic data that you can collect. However, different types of diagnostic data are often collected in different ways. In fact, there are several different trace facilities available on i:

• Job Trace
  Job trace (STRTRC) traces the program flow (among other things) within a job. I blogged about job trace some time ago in the article IBM i Job Trace. The Trace Job (TRCJOB) command still exists, but it now invokes STRTRC automatically. A little secret about Job Trace is that it uses Performance Explorer trace under the covers for the data collection; however, job trace does much more than just provide a customized interface over Performance Explorer.
• Performance Job Trace
  Start Job Trace (STRJOBTRC) is a job trace function to collect performance statistics. As of the 6.1 release, it invokes STRTRC.
• Performance Explorer
  Performance Explorer trace (STRPEX) is used to collect very detailed data, primarily intended for performance diagnostics.
• Licensed Internal Code (LIC) traces
  LIC traces can be started by going into System Service Tools (SST) or Dedicated Service Tools (DST) and starting the LIC trace service function. The Trace Internal (TRCINT) command is a command interface to LIC trace that alleviates the need to go into SST or DST to start or manage your LIC traces.
• Trace Connection
  The Trace Connection (TRCCNN) command, which provides a trace similar to a communications trace, actually does a LIC trace under the covers, so TRCCNN will start a TCP/IP or Sockets LIC trace.
• Performance Trace
  Start Performance Trace (STRPFRTRC) collects multiprogramming level (MPL) and transaction LIC trace data. It actually invokes TRCINT to start a LIC trace.
• Communications Trace
  Communications Trace (STRCMNTRC) is used to collect trace data as it flows over communications lines.
• Trace TCP Application
  Trace TCP Application (TRCTCPAPP) is a specialized interface for starting traces specifically for a set of TCP/IP Application Servers. Of interest is that you can also start additional traces when you trace a TCP/IP application server, such as job traces or LIC traces.

As you can see, there are a wide variety of interfaces to collect trace data, but it really comes down to four basic types of trace data collection:

• Job Trace
• Performance Explorer
• LIC Trace
• Communications Trace

The Work with Traces command can allow you to manage any one (or all) of these traces. It has one parameter – the Trace (TRC) parameter – which allows you to specify which traces you want to work with. Supported values are:

• *ALL – work with all traces
• *CMN – work with communications traces
• *JOB – work with job traces
• *LIC – work with LIC traces
• *PEX – work with Performance Explorer traces

When you first go into the WRKTRC interface, you will see the traces that are applicable based upon the TRC parameter. If there are no traces active, a message will be displayed stating that no traces are found. What's particularly useful with WRKTRC *ALL is it will show you all the traces that exist on the system in some form (active or ended); it is not uncommon to start a trace but forget that it is running; WRKTRC can help you determine if you have forgotten traces that are still running. Or perhaps you have a trace you started and ended, but never deleted (to free up storage); WRKTRC will show you those traces.

The F6 key allows you to start a new trace. You can then select the type of trace you want from the menu options listed. The appropriate command prompt interface for starting your selected trace is then displayed.

In the following screen capture, you will see an example of working with traces with one trace of each type active:

 

 

The following screen capture is an example of working with traces when there is a trace of each type that has been ended:

 

 

For any trace, you can take actions that are appropriate for that trace type and the state that it is in. Details of what actions are appropriate for what trace types can be found in the online help. The online help also describes the various statuses that can occur for each type of trace.

Finally, there are some additional diagnostic trace capabilities that exist on i that are not supported with WRKTRC; this is because these trace types are not commonly used any more. For example, Trace CPI Communications (TRCCPIC) and Trace ICF (TRCICF) are related to SNA communications functions. User Trace (xxxUSRTRC) is replaced with Job Trace.