Clone an Integrated Windows Server

In a Windows server environment, it's normal to have a need to quickly create another Windows server to be used for a new application or even as a backup system to test the latest service pack without interfering with your production server. The problem though is creating multiple servers from scratch gets tedious when needing to install additional device drivers, service packs and common applications after the operating system is installed. But with a recent new-function PTF, support for cloning integrated Windows servers is possible.

An easy-to-use GUI Wizard duplicates an existing iSCSI based Windows 2003 or Windows 2008 server by copying the network server description (NWSD), network server storage (NWSSTG), virtual Ethernet line descriptions, and then makes them ready for that first activation. This support was added to IBM i 7.1 in December 2010, and IBM i 6.1 in May 2011.

Although it's possible to duplicate a single server for test purposes, the primary reason cloning support was added was for rapid server deployment where multiple blades or System x servers are designated for Windows server installations. The first task in creating servers based on an existing server is to ensure your base server is ready to be cloned. This means your server is fully installed with a supported Windows server operating system, latest service packs and any applications that are needed on the clone servers. Once the base server is ready to be cloned, a Windows tool named “SYSPREP” is used to remove server specific information such as log files and license key information and prepares it for a first-time run welcome experience. Information on how to access SYSPREP on Windows 2003 or Windows 2008 is available from the following link: http://publib.boulder.ibm.com/infocenter/toolsctr/v1r0/index.jsp?topic=%2Fdpsccm%2Fdpsccm_t_running_sysprep.html

The SYSPREP command suggested for the integrated server environment should be run as follows to prepare the cloned server to run the mini out of box experience when first started.

• Windows 2003:  SYSPREP -reseal -mini
• Windows 2008:  SYSPREP /generalize /oobe

After SYSPREP has been run, shut down the server and start making as many copies as needed. To start the server cloning wizard, use the Web GUI, select Create Server task and select the Base on an existing object option and specify the NWSD name of the server to be cloned as well as the name of the new server. Continue through the wizard to copy the storage, specify a compatible remote system, and then click Finish on the Summary page. The progress bar will show that status of the creation process, which will take some time duplicating large storage spaces.

The clone server is created with the same properties as the base server with one exception. The virtual Ethernet point-to-point connection for the clone server will have a different TCP/IP interface on IBM i and will need to be manually set on Windows once the server is activated to be in the same subnet.

For detailed instructions on the entire cloning process and using the integrated server cloning wizard, see the section titled “Server cloning road map and checklist” in the IBM i iSCSI Solutions Guide. 

The required PTFs for this support are as follows.

• IBM I 6.1 PTF SI43599 and WebNav GUI service pack PTFs (WSP8)
  or IBM I integration with BladeCenter and System x Group PTF SF99357 Level 20.
• IBM I 7.1 PTFs SI41049, SI41817 and WebNav GUI service pack PTFs (WSP3)
  or IBM I integration with BladeCenter and System x Group PTF SF99369 Level 4.

I'd like to thank Thom Haze from the Integrated Server Development team for writing this blog article.

 

 

 

IBM Systems Director Navigator for i Work Management Integration with Performance Tasks

In last week's blog on the features of the green screen “Work with Active Jobs” interface, there was a comment regarding the nice features of the graphical tools (System i Navigator or IBM Systems Director Navigator). I had written a prior blog on some of the “Cool Things with the Work Management GUI” that covered some additional function within the work management GUI. In that blog I wrote briefly about the enhanced integration between the Active Jobs GUI and the Performance GUI. 

I'd like to expand upon that...

Active Jobs (or WRKACTJOB on the green screen) is often the first place you should look if you want to have some basic understanding of the system's performance characteristics and how the work is running. If you discover some job that appears to be using a lot of system resources, whether CPU or I/O, you may want to dig deeper to understand more about what’s going on. 

In the 7.1 release, there’s much greater integration between the Active Jobs tasks and the Performance tasks that allows you to go from working with active jobs directly to analyzing performance data.

For my examples, you’ll notice I’ve customized the columns in my Active Jobs view–I added the metric “Disk I/O Count,” sorted on the CPU % column, and reordered the columns so that “CPU %” and “Disk I/O Count” were easy to see and include in my screen captures. This type of flexibility with the GUI is great to customize what you need to see.

I can see from Active Jobs that the job QZRCSRVS with current user dmmay (Hey! That's me!) had been using CPU, but also that it had been doing disk I/O. We can use this job as my example job for showing how you can get from an active job to the performance data for it. By clicking on the little arrow to the right of the job name, you bring up a pop-up menu of the options to work with that job; the performance tasks are nested under the performance option, as my screen capture (below) shows.

The performance options to work with Collection Services data are only available when Collection Services is active, which it should be as that’s the default behavior beginning in 6.1. When you select these options to work with performance data from the Active Jobs interface, the most recent Collection Services collection will be used.

If you select the option to Investigate Job Wait Data, you’ll go from the Active Jobs interface to a new tab in your Navigator session to display the Collection Services job wait data for the job. You’ll see something similar to the chart below.

But, in my example, the timeframe is too large and I want to focus in on the data at the current time.  I can do that by zooming in on the chart to display the data more closely.  (At some point, I'll also write more on how to use the Performance Data Investigator).

By zooming in on the performance data on the very right-hand side of the graph, I can see the Collection Services data in much more detail, and I discover that my job has been page faulting, see figure below.

 

With Collection Services data, you can't get more detail about why the job was page faulting. 

However, going back to the Active Jobs interface makes it easy for you to get the detailed data you need to investigate the problem further.

If you have the IBM Performance Tools licensed program product, option 3 (Job Watcher) installed, you will have the ability to start Job Watcher from the Active Jobs interface, again, under the Performance options for that job.

This will launch the Job Watcher wizard to lead you through starting a Job Watcher collection. Once you have the more detailed performance data you can get with Job Watcher, you can use the performance tasks to further investigate the data. Note, however, that you’ll need to reproduce the performance problem while Job Watcher is active in order to have the desired detailed data for analysis.

The steps I just went through were how you can get to performance data for one job by starting from Active Jobs.

You can also look at system wide performance data by starting with Active Jobs.  To do this, you don’t select any particular job to work with, but rather use the action pull-down to select the option to Investigate Jobs Wait Data (you can also Investigate Jobs Statistics Data as well). See figure below.

Displaying the Jobs Wait Data will display the Collection Services job wait data for all jobs, as the following screen capture shows.

 The Job(s) Wait Data show the details of what the job had been waiting on. Refer back to the blog on “i Can Tell You Why You're Waiting” for more information on waits.

The option to Investigate Jobs Statistics Data will present Collection Services jobs statistical data in table form, where you can search and sort, customize columns, etc., to work with the performance data in table form. By default, all columns are displayed in the table, and you must scroll right or left to display them. Just like any other table view in IBM Systems Director Navigator, you can customize which columns you want to see, the order in which the columns are displayed, as well as the sort order you want to use. By looking at Jobs Statistics Data, you’ll discover all sorts of job metrics in Collection Services that may be interesting for analysis purposes.

Work With Active Jobs

This week's blog was written by Mike Russell, who is a member of the IBM i work management team. Everyone knows about the Work with Active Jobs command, but Mike shares some of the lesser known features as well. Thanks, Mike!

This article describes the Work with Active Jobs command. Work with Active Jobs helps you see and manage what is running on your IBM i system. It shows a list of all the active jobs with their current status and basic performance information.

Basic Features

The job type, current user, priority, memory pool, status and number of threads can all be shown by pressing F11. Basic performance information shows the job’s processor usage, paging and interactive response time. Options can be entered to control the job or find more detailed information about a single job. F23 can be used to see all the options available and F24 to see all the function keys available.

For information that can be different for each thread in a job, the initial thread’s information is shown. The status shows whether the job is currently running or not. The status values ending in W show the job is waiting. The same value ending in A means it is waiting in the activity level.

Elapsed interval

Work with Active Jobs gathers current information for each active job when the command is run and each time the screen is refreshed. The current performance information is compared to information collected at an earlier start time. The difference between the current time and the start time is called the elapsed interval. The elapsed interval is used to calculate the CPU percent, number of interactions, average response time and auxiliary I/O operations for each job. The elapsed interval is shown at the top of the screen along with the percentage of CPU used by the system and number of active jobs. When the WRKACTJOB command is used for the first time in a job, the elapsed interval is zero and no performance data is shown. This is the start time of the elapsed interval. The F5=Refresh key causes the end time of the elapsed interval to be set to the current time, the start time remains the same. The F10=Restart key causes the end time of the elapsed interval to be set to the current time, with the start time set to the previous end time. In other words, F10 shows you what has happened since the last time you looked. F5 extends the elapsed interval. F13=Reset statistics can be used to set the elapsed time to zero and reset the display.

If you are looking for performance problems, you want to use F10 occasionally. With a long elapsed time the performance information becomes watered down, making it difficult to see what's currently happening.

Other Features

Sorting and limiting

It can be difficult to find what you are looking for in a long list of jobs. There are features in WRKACTJOB to help you find, sort and limit the number of jobs shown. F7=Find displays a pop-up to allow you to enter a search string and what column to search. F8=Repeat find will find the next instance. F18=Bottom and F17=Top will get you to the bottom or top of the list without scrolling.

By default the jobs are grouped under the subsystem they are running in. You can sort the jobs by the values in any column on the display by moving the cursor into the column and pressing F16=Resequence. The SEQ (sequence) parameter on the command can also be used to sort the jobs. And the F20=Subset pop-up also lets you change how the jobs are ordered. Sometimes sorting the jobs into a different order can make it easier to see which jobs are using the processor, work with all jobs in MSGW status, or monitor the paging of jobs running in a memory pool.

By default all active jobs are shown except those that are normally expected to be idle. Idle jobs include the suspended system request and group jobs, disconnected jobs, and prestart jobs that are waiting for a request. You can show the idle jobs by pressing F14=Include. There are several parameters on the command to let you work with a smaller list of jobs. The CPUPCTLMT and RSPLMT parameters let you only see jobs that are using the processor or having long response times. The SBS parameter lets you see only jobs in subsystems you are interested in. The JOB parameter lets you see only jobs matching a generic job name or system jobs or subsystems. The F20=Subset pop up lets you change which jobs are included on the display without exiting the display and entering the command again.

Option 9=Exclude can be used to remove jobs from the display that you're not interested in. This allows you to get the jobs you are interested in on the same screen so you don’t have to roll back and forth to compare their performance information. The jobs remain excluded until you use F13 to reset the display. F21=Nondisplay instructions/keys can be used to get more jobs shown at one time. The option and function key descriptions are removed. F21 again will change back to display the descriptions.

Automatic refresh

WRKACTJOB can be left running to monitor the system activity by pressing F19=Start automatic refresh. The screen will be refreshed every 5 minutes. You can change how often the screen is refreshed with the INTERVAL parameter. In release 7.1, the automatic refresh function was changed to only show activity for the latest interval. In previous releases, the elapsed time increased each interval to show activity for the entire period since automatic refresh was started, making the information less and less useful over time.

Setting the Function field

The function field usually shows a high-level command, menu or program used by the job. You can use the Change Current Job QWCCCJOB API to customize the function field in a long-running program to something more meaningful. USR- will precede the specified value on the display.

 

Multi-Threaded Interactive Jobs?

A comment was posted to last week's blog on IBM i Job Structures. It read:  

Dawn:
This job structure article jogged my memory regarding a recent situation which I witnessed with a Series i customer. They have multi-threaded INTERACTIVE jobs running on their system. Can you sometime explain the concept of multi-threaded interactive jobs? I know and understand multi-threaded batch jobs but know nothing about multi-threaded interactive jobs. How are they created? What purpose do they serve? From a performance standpoint, does QPFRADJ adjust activity levels for multi-threads of interactive jobs? Any additional info would be helpful. Thanks!

An IBM i programmer cannot create secondary threads in an interactive job; the allow multiple threads parameter on job descriptions is ignored for interactive jobs and you will receive an error if you attempt to run a program that starts secondary threads when in an interactive job. This is because interactive jobs are not thread safe. One reason interactive jobs are not thread safe is because I/O to display devices is not thread safe. 

So how can a multithread interactive job be created? 

Threads were introduced to the system in V4R2. In V4R4, the threads support was extended to add “system threads.” System threads are threads of execution used for system functions and can be created in any job, including interactive jobs and jobs that have the allow multiple threads job attribute set to *NO. Because system threads can only be used by operating system developers, there are expectations for how those threads are used to ensure thread safety issues do not occur. Java uses system threads and several DB2/SQL functions in the operating system also use system threads for parallel operations.

You'll find no documentation in the IBM i Information Center on system threads. There are bits of documentation, however, that will give you small clues about them. For example, the command help for the allow multiple threads parameter includes the following statement: “Specifies whether or not the job can run with multiple user threads. This attribute does not prevent the operating system from creating system threads in the job.”

Next week I'll write about threads and the performance adjuster.

 

IBM i Job Structures

Last week, I wrote about detaching spooled files. As I was writing that article, I started to describe the IBM i job structures. It became apparent that there was too much information for that one blog, so this week I'll review IBM i job structures.

There is little formal documentation on job structures. You can find bits and pieces in the Information Center if you look at the documentation for the Display Job Table (DSPJOBTBL) command or the Create Job Structures API. The Display Job Table online command help ('green screen' help) probably has the most information regarding job structures.

Every job on the system has three main pieces that I generally refer to as “job structures”:

• an entry for the job in the job table
• permanent job structures
• temporary job structures

The “job table” is also called the “work control block table” and is the primary data structure used to keep track of all jobs in the system. Jobs on job queues, active jobs and ended jobs with spooled output all consume an entry in the job table. The job table is of limited size and can hold approximately 485,000 entries; thus the upper limit on the Maximum Number of Jobs (QMAXJOB) system value. It's not a good idea to run near the maximum number of jobs, as I described last week. The system job table should be kept to a reasonable size, and if you generally have a consistent number of jobs, the system will automatically do this for you.

The system has a built-in notification mechanism for when the job table is nearing its maximum capacity, as configured by the QMAXJOB system value. By default, the system will send the CPI1468 message (“System job tables nearing capacity”) to the QSYSOPR message queue (and to the QSYSMSG message queue if it exists) when the job tables are 90 percent full. IBM has provided the capability to customize the percentage at which this message is sent. PTFs for 5.4 (SI29585) and 6.1 (SI30171) provide this support (no PTFs are needed for 7.1 as this support is part of the operating system). You create a data area named QMAXJOBPCT in QSYS to control the percentage at which you want the warning message to be sent. See the PTF cover letters or the redpaper referenced at the end of this blog for the detailed instructions.

If you have a 'run-away' job situation occur where you have many more jobs than is typical, you may want to consider compressing the job table. This is an action that is done at IPL and can be long-running. You should only do this when necessary. The system IPL attribute of Compress Job Tables controls this; the default value is *NONE and is recommended. If you have a need to compress the job tables, use the value of *NEXT. The Rochester Support Center has a couple knowledge base articles regarding job tables and job table compression that you may find helpful:

• Slow IPLs and Job Table Compression
• Job Table Capacity

The permanent job structures are as the name implies -- permanent and survive IPLs. These job structures are what allow ended jobs with output to remain on the system until the output is deleted or detached from the job. The permanent job structures consist of three main pieces:

1. the spool control block
2. the job message queue (for job log messages)
3. the local data area

The temporary job structures are temporary, do not survive IPLs and are only associated with a job when it is active. There are many parts to the temporary job structures and they include such things as:

1. the process control space
2. the qtemp library for the job
3. the job performance data space for performance data
4. … and several other structures

The permanent and temporary job structures are created ahead of time and managed by the system. They are recycled for reuse; that is, once a job ends, the temporary job structures can be used by another job. Once the job has left the system (ended and no spooled files associated with it), the permanent job structures can be used by another job.

The Total Jobs (QTOTJOB) and Active Jobs (QACTJOB) system values define the initial values for the total permanent and temporary job structures. The Additional Total Jobs (QADLTOTJ) and Additional Active Jobs (QADLACTJ) system values define how many additional permanent and temporary job structures are created when necessary.

As I wrote last week, ended jobs with spooled output continue to use a job table entry and the permanent job structures for that job. Detaching spooled files can free up those structures for use by other jobs and reduce the number of jobs on the system.

Jobs can also have their job logs in an pending state (job log pending). I haven't yet written about job log pending, but jobs with pending job logs also consume an entry in the job table entry and a permanent job structure. Good job log management can reduce the number of jobs on the system and I will write more in the future about job log management.

The redpaper “Best Practices for Managing IBM i Jobs and Output” has more information on this topic along with recommendations for setting the system values to control job structure management by the system, considerations for job table compression, more information on the job table warning PTFs, as well as suggestions for reducing the number of jobs on the system. This redpaper is one I consider a “must read” for all IBM i administrators.

 

 

Detach Spooled Files From a Job

I'd like to thank Kevin Vette for his assistance in writing and reviewing this week's article. Kevin works in the IBM i development lab on the print and spool team.

 

Many of the larger IBM i environments need more sophisticated methods to manage the number of jobs on the system. If the absolute maximum number of jobs--as defined by the Maximum Number of Jobs (QMAXJOB) system value--for the system is reached, there can be some negative results such as:

• An unplanned outage due to hitting the maximum number of jobs
• A large numbers of jobs that contribute to long IPLs for unplanned outages.
• Consumption of job table entries

The main problem many customers have with reaching the maximum number of jobs is not with active jobs but with jobs that are in OUTQ status. In other words, the job ended but it still has at least one spooled file, perhaps only the job log. 

A job on the system has control structures associated with it, even when the job has ended, if it has spooled files. These structures include an entry in the job table as well as the permanent job structures to store information about the job. 

IBM has provided a system value Spooled File Action (QSPLFACN) and a job attribute (SPLFACN), which can help reduce the number of jobs on the system. The Spooled File Action values are either *KEEP (the default), to keep the spooled files associated with the job, or *DETACH, which allows the spooled files to be disassociated with the job. By detaching the spooled files, the job's control structures can be freed for reuse by other jobs while the output remains available. Detaching spooled files can result in fewer jobs on the system, which is a good thing. 

If you have jobs that have ended but have kept the spooled files, using the Change Job command (CHGJOB) with the SPLFACN(*DETACH) parameter will allow the spooled files to be detached and the job to go away.

Detaching the spooled files from the job may cause some operational differences since the spooled files can no longer be found using WRKJOB, CHGJOB, and so on. They can, however, be found using the WRKJOBLOG (for job logs) or WRKSPLF (using the JOB parameter) commands. All the normal spooled file commands like DLTSPLF work the same as they do on spooled files that are not detached.

One thing should be noted about the *DETACH support. While unlikely to happen, it is possible that if spooled files are allowed to exist on the system for long periods of time after the job ends, another spooled file might get generated which has the exact same spooled file name, job name, job user, job number and spooled file number. Many applications use only those parameters or just some of them to reference a particular spooled file. It would thus be possible for those applications to fail with an error for a duplicate spooled file (or duplicate job) message. All spool interfaces have added additional parameters for the job system name (the system where the job that created the spooled files ran) and create date and time so that the correct spooled file can be selected but older applications may need to be brought up-to-date to use them. 

For more information about detaching spooled files and managing jobs on the IBM i, take a look at the redpaper Best Practices for Managing IBM i Jobs and Output. 

 

A Simple Example:

In this example, I had a job that had two spooled files associated with it. The job log no longer existed because the system had automatically cleaned up old job logs. To free up the job structures, I detached the spooled files from the job. From the example, you can see that WRKJOB does not find the job after the spooled files are detached, but WRKSPLF will.

 

WRKJOB JOB(453851/DMMAY/QPADEV000D)

option 4 to work with spooled files:

 

CHGJOB JOB(453851/DMMAY/QPADEV000D) SPLFACN(*DETACH)

 

WRKJOB JOB(453851/DMMAY/QPADEV000D)    

Job 453851/DMMAY/QPADEV000D not found. 

 

WRKSPLF SELECT(*ALL) JOB(453851/DMMAY/QPADEV000D)

 

 

The Secret History Log Enhancements

This week, I thought I’d write about some enhancements that have been made to the history log in recent releases. While these changes have been available for some time, they are well hidden because they were generally small changes on existing interfaces. The history log, as we all know, can be very valuable for understanding what has happened on the system, but it can also be a little overwhelming to deal with because of the potential volume of messages that get logged.

 

History Log Size - *DAILY

In V5R4, we made a little enhancement to the History Log Size (QHSTLOGSIZ) system value. A special value of *DAILY was added. This option allows for a new version of the history log to be created each day, rather than based upon size. If you have a busy system and reach the maximum number of records in a day, you will get a new log version when that occurs, so on these busy systems, you may have more than one log version a day even if you specify *DAILY for the system value.

I'll make an observation about the values allowed for the history log size. IBM i allows you to specify a size of 1. I have no idea why this is, but it’s silly! In fact, if you do specify 1 for the QHSTLOGSIZ system value, the system actually uses 10 – which is still ridiculously small. The default size is 5,000, and that is also too small for today's environments! I like the capability to have history logs created on a daily basis and that approach eliminates the need to play around with log size values.

 

Open List of History Log Messages API

In the 6.1 release, we added the Open List of History Log Messages (QMHOLHST) API. The API provides a programmatic interface to access history log messages. Prior to the 6.1 development of this API, your choices for accessing history log messages were either through the Display Log (DSPLOG) interface or directly by reading the history log files.

There are several advantages to using the API:

• It’s a formal programming interface; prior to the introduction of the API, you had to read from the files directly to programmatically access the history log messages.
• You can retrieve many messages with one invocation of the API, as opposed to reading the files record by record
• Using the API, you can filter on message severity or you can filter on the message type; in addition you can specify with the filters to either include or omit the filtered data
• You can also identify specific message IDs that should be retrieved or omitted
• This API can be used to get messages from the QHST message queue into the history log files. Sometimes you need to do a DSPLOG twice in order to see the latest messages, but that is not the case when using the API.

 

Watches

I wrote a blog on watches some time ago. Using watches you can watch for messages that are sent to the history log. But be careful! Do not watch for common messages (such as job started/job ended messages) since those will be matched frequently. If you do need to watch for a commonly sent message, qualify it with a comparison data so the match criteria are more selective.

If you have a need for real-time notification of messages sent to the history log, using watches is more effective than using the API or the DSPLOG command. Note that while Management Central has message monitors, you cannot monitor the history log messages with these message monitors; rather, you use file monitors to monitor the history log through Management Central. However, file monitors are more resource intensive than watches.

 

Print Second Level Message Text

Way back in V5R3, support was added to the Display Log (DSPLOG) command to allow you to print the second level text of messages from the history log. This was done by extending the keywords on the Output (OUTPUT) parameter to support *PRTSECLVL.

 

 

 

 

Avoid Locking Libraries in the Library List

There is a system value, Library Locking Level (QLIBLCKLVL), that you can use to control whether libraries that are in a job's library search list are locked. This system value is not new – it has been around since the V5R1 release – but it seems that it’s not very well known as I've talked with several people who’ve never heard of it.

The Library Locking Level system value has the default value of '1', which indicates that the libraries in the library search list are to be locked in each job. Each library in a job's library search list is locked when the job is started and the locks are released when the job ends.

As the number of active jobs on a system grows, the number of locks just for libraries can get very large. Consider an example where there are 1,500 user jobs and the library search list has 10 libraries – that's 15,000 locks held on the system just to protect those 10 libraries. This is not an efficient use of the system's locking table. This affects the performance of job initiation and termination when the locks are allocated and released; it also has an effect on overall system performance as there is a single lock table and additional overhead as this table grows very large.

The “Best Practices for Managing IBM i Jobs and Output” Redpaper documents IBM's recommendation of setting this system value to 0 to not lock the libraries in the library list.

While IBM recommends setting this system value to 0, to not lock the libraries, it could not make that value the default value because, without additional changes, setting this system value to 0 would allow a library to be deleted or renamed even when the library is in the library search list.

If you change the QLIBLCKLVL system value to 0, you need to take an additional step to prevent the libraries from being deleted or renamed—you just need one job to allocate the libraries you want to protect. This job can be set up as an autostart job that runs in the controlling subsystem; it would simply do an Allocate Object (ALCOBJ) on each library that needs to be protected, then loop on a Delay Job (DLYJOB) command. This job would remain active until the controlling subsystem is ended or the job is explicitly ended.

By using this strategy, you can protect the libraries from being deleted or renamed and you can greatly reduce the number of locks on the system.

One final point - the QSYSARB system job locks the libraries that are defined in the system library list (QSYSLIBL) and the user library list (QUSRLIBL) system values. IBM has a support article on this. 

 

 

 

Automated Problem Reporting With Service Monitor

IBM i (and its predecessors) has had the capability to automatically identify and report software problems to IBM for many releases. This was first introduced as the “Software error logging” (QSFWERRLOG) system value.

The capability to automatically identify problems when they occur is something we call First Failure Data Collection (FFDC). The intent is the first time a problem occurs, the data necessary for problem determination is automatically collected. That data, for operating system problems, can then be sent to IBM for diagnostic purposes. The lofty goal is to identify and resolve problems the first time they happen without ever having to recreate the problem.

This is an admirable goal, but in reality is very difficult to achieve. The challenge is that defects occur in unknown and unexpected places –– simply because defects are unexpected failures and programmers don't intend for unexpected failures to occur. As such, the capability to programmatically identify what data should be collected for a potential software bug is inherently flawed as it requires the ability to predict knowledge about what the problem may be in order to program the data collection routines.

In the 5.4 release, we decided to take a new approach to our FFDC support; key to that new approach is the recognition that one can never predict where a problem may occur. As such, there needed to be a dynamic way to modify the parameters around the identification of potential problems and the data that is collected for those problems –– this new support for software problem reporting was called Service Monitor. With the introduction of Service Monitor in the 5.4 release, we moved to a design that is dependent upon a “policy” that identifies potential problems along with the data that should be collected for those problems; this policy can be updated dynamically. Service Monitor is automatically started and supported with the *LOG option of the QSFWERRLOG system value.

While the Service Monitor has existed for a few releases now, it seems that it is a relatively unknown feature. The biggest reason is probably due to the fact that there’s no high-level overview of the Service Monitor function within the Information Center. Bits and pieces of information can be found by reviewing APIs, but there’s no summarization of the function and its capabilities.

Service Monitor, then, is a policy-based software function that is used to automatically identify problems that occur and to take the defined actions defined by the policy when the problem occurs. It’s primarily used for problems within the operating system and licensed internal code.

In the 5.4 release, customers primarily noticed the presence of the Service Monitor function by the many QSRVMONxxxx jobs that run in the QUSRWRK subsystem; a job started for each potential problem that could be reported to IBM. A common question I've heard is “What are all those jobs for?” In the 6.1 release, the design was changed to use prestart jobs rather than individual jobs, so the number of jobs has been reduced, but the Service Monitor function remains.

The policy file that is used by Service Monitor is maintained by the IBM Support Center in Rochester. As experience is gained with known problems, or when new problems are encountered, the policy file can be updated. The latest version of the policy file is downloaded to your system when you connect to IBM using Electronic Service Agent.

The policy file identifies the symptom of the problem, which can be a message, a licensed internal code (LIC) log (also known as a Vlog), or a product activity log (PAL) entry. The policy also identifies the action that is taken when that symptom occurs –– that action could be to collect diagnostic data to send to IBM, download a PTF that corrects the issue or some other action.

Several months ago I wrote a blog about Watches. Service Monitor uses watches as the underlying mechanism to implement the automated notification mechanism for problems that can be detected. Service Monitor policies identify the conditions to watch for –– which can be messages, LIC logs and problem activity logs (PAL entries) –– and thus Service Monitor sets up a watch for each item in the policy file. When the watch condition is matched, Service Monitor has an exit program that gets invoked and uses the policy definition as the way to identify the actions that should be taken when the watch occurs.

If you use the WRKWCH command to look at the *SRVMON watches, you will find many active watches (unless you have changed the QSFWERRLOG system value to *NOLOG). By displaying individual watch entries, you can see the kinds of things that service monitor watches for –– messages, LIC logs or PALs. For example, on a 6.1 system, if you look at the SRVMON0003 watch, you can see that it’s monitoring for CPF1101 sent to the QSYSOPR message queue. CPF1101 is “Subsystem &1 had a function check.” You can imagine that IBM probably wants to know if a subsystem takes a function check and has some basic diagnostic information that should be reviewed if this situation occurs. When this message occurs, the policy indicates that the job log should be collected and the problem reported to IBM.

Most of the policies used by Service Monitor are for LIC logs. Since LIC logs are commonly used to log diagnostic information for problems detected by the Licensed Internal Code, LIC logs are something easy to monitor for and then to automatically send the log data to IBM for review.

 

PTFs for Additional Work-Management Controls

The work-management team recently made available some changes with PTFs to help control the behavior of job creation for cases when a program may get in a loop submitting jobs. The change made with the PTF is to slow down the creation of new jobs when the system is approaching the limit for the maximum number of jobs (QMAXJOB system value). 

QMAXJOB defines how many jobs can be on the system and includes all jobs: jobs submitted to a job queue, active jobs and ended jobs with spooled output. When the system reaches the limit specified by QMAXJOB, no more jobs can be started and the system can become inoperable. 

There’s a warning message--CPI1468 “System job tables nearing capacity”—that’s sent to QSYSOPR when the system is nearing the limit of the maximum number of jobs. Support was added to customize when these messages are sent with the following PTFs:
•    6.1 - SI30171
•    5.4 - SI29585
•    This same function exists in 7.1 without additional PTFs

By default, the CPI1468 is sent when the job tables are 90-percent full. In some situations, that’s too late and you don’t have time to recover from the condition. Create a data area named QMAXJOBPCT in QSYS with the desired percentage value at which you want the message to be set. The PTF cover letter has more detail.

The new PTFs that slow down the creation of new jobs when the system is approaching the QMAXJOB limit are:
•    7.1 – SI40568 
•    6.1 – SI40569
•    5.4 – SI40571

With these PTFs, the system adds delays to starting new jobs when the number of jobs is at 90 percent of the QMAXJOB setting. Override this percentage by creating a data named QWTSBSDLY in QSYS; for example, you could have the threshold be set to 80 percent. There are no delays if the number of jobs in the system remains below that percentage.

The delays start at a half second, but are increased as the system approaches the QMAXJOB limit. A single delay isn’t noticeable, but if you have a job that’s looping starting jobs, these delays can add up and offer some protection to keep the system operational as you approach the QMAXJOB limit. You can still run out of job-table entries, but it will take longer and you’ll have a little more time to react to the situation.

In 2008, we wrote a Redpaper, “Best Practices for Managing IBM I Jobs and Output (and a few other special tips)." This Redpaper covers general recommendations for improved management of jobs and output. Provided in that Redpaper is a sample CL program source for a Submit Job (SMBJOB) exit program. The submit job exit program is the best way to control the runaway submission of many jobs.

Another set of PTFs were provided that change the default behavior of the system to briefly display the end-of-job display when an interactive job ends. Previously, when an interactive job was ended, the end-of-job display would remain until the user pressed enter or 30 seconds had passed.  With these PTFs, this delay is removed and the interactive jobs are ended immediately without delay due to the end-of-job message; the message is still sent, but is only seen briefly. This change is helpful for ENDSBS, for example. Prior to the PTFs, ending an interactive subsystem would generally not end the jobs right away because the end-of-job display would be displayed for 30 seconds for those jobs where the user didn't press enter. With these PTFs, the jobs are ended right away.

These  PTFs are:
•    7.1 – SI40867
•    6.1 – SI40866 

The PTF cover letters containing documentation on how to create a data area named QWTEOJDSP in QSYS to get the old behavior back.