QTEMP – Temporary or Permanent Storage?

I received a question regarding the QTEMP library, prompted by last week's blog on Understanding Disk Space Usage. I thought sharing the question and providing the answer in this week's blog would be helpful for everyone, as well as expanding upon how the system views and manages QTEMP when it comes to disk space usage.

The email I received was from John Knox:

When it comes to temporary storage, the value in the job run attributes in WRKJOB only shows temp storage used by an application, presumably for working storage, etc. What it doesn’t show is space used in the QTEMP library for that job. Several times this has been an issue for us because an application creates a temporary file or user space in QTEMP and the file gets very large and we have a difficult time determining which job is the offender. My intention was to create a utility to scan all active jobs and display a screen showing the sizes of QTEMP for each job. However after much research there doesn’t appear to be an easy way to query the size of QTEMP libraries for the jobs on the system unless there is an API I haven’t found yet.

So my real question is, do you know of any way to query the size of QTEMP for each job on the system?

First, let me explain how the system accounts for storage in the QTEMP library.

The QTEMP library itself is created out of permanent storage; any objects that you create in QTEMP --work files, data areas, message queues, or whatever -- are also permanent objects. When the job ends, the system deletes the objects in the QTEMP library. Because QTEMP itself and the objects created into QTEMP are permanent objects, the storage they consume is tracked as permanent; it is charged to the user profile and the objects in QTEMP can be manipulated like any other object -- saved and restored, renamed, moved to other libraries, etc. The library name, QTEMP, reflects the fact that it is associated with a job only when the job is active. Perhaps it would have been better if IBM would have called QTEMP the “QJOB” library.

So back to the question asked - how can an administrator determine the size of QTEMP libraries for the system and determine if there are jobs with QTEMP libraries consuming a lot of disk space?

There are a few ways to do this:

1. You can display your own job and view your own QTEMP. For your own job, you can do a DSPLIB QTEMP OUTPUT(*PRINT) which will give you the total size of the library at the bottom of the listing.  You can use the Retrieve Library Description API to get the size of your job's QTEMP library.
   
   
2. You can work with an active job (WRKJOB) and look at the job's library list, which will include QTEMP. You can then display the contents of the QTEMP library; this is a manual approach and you have to calculate the total size from the displayed list.
   
   
3. You can use the Open List of Objects API to get a list of objects in another job's QTEMP library.  You need to call the API using optional parameter group 1 and specify the other job. The program would also need to sum the sizes of all objects returned to get the total size.
   
   
4. Extending upon option 3, you can use the Open List of Jobs API to get a list of all active jobs and then for each job, use the Open List of Objects API for QTEMP. Be aware that the jobs can start or end after the Open List of Jobs API returns its results.
   
   
5. You can use the Call Job Interrupt Program Exit Program to run a program in a different job which could be a program to determine the size of the QTEMP library for that job. I'll write more about this exit program in next week's blog.

 

 

 

 

Understanding Disk Space Usage

This topic is one I know many i users are very familiar with, but I get questions often enough about managing disk space that I thought a review of the parameters controlling disk space consumption and managing disk space growth would be a worthwhile topic.

Disk space that is used by i is categorized into two major types:

1. Permanent. This is the storage that is used that survives IPLs. Objects that you create are permanent objects. Permanent disk space is tracked by user profile and the Maximum allowed storage (MAXSTG) parameter on the user profile controls the upper limit of disk storage that an individual user profile can consume. Note the default value is *NOMAX.

2. Temporary. This is the storage that is used for temporary objects and internal data structures and does not survive IPLs. Temporary disk space is tracked to the job that allocated the storage and the maximum value comes from the Maximum temporary storage (MAXTMPSTG) parameter from the class object used when the job is started. Note the default is *NOMAX.

IBM made a change in the 7.1 release that jobs that exceeding their temporary storage limit will be held rather than ended. This was described in the blog article, IBM i 7.1: Jobs Exceeding Their CPU or Storage Limits are now Held. 

Controlling disk space consumed is important as the system can terminate unexpectedly if all the storage is used. The system values Auxiliary storage lower limit (QSTGLOWLMT) and Auxiliary storage lower limit action (QSTGLOWACN) define the low storage threshold and the action taken by the system when that low storage threshold occurs.

However, if you are having an issue with storage consumption, the major challenge is identifying where the storage is going quickly enough to resolve the problem before it becomes a major issue.

For tracking permanent storage, you should have a routine process in place where you understand the size of your permanent objects. Objects such as the history log, journal receivers, spooled output, performance data and several other types of storage may not need to be kept online on an ongoing basis; you should have a process where you save off the critical data and then delete the unnecessary data from your system.

The system provides the Retrieve Disk Information (RTVDSKINF) and Print Disk Information (PRTDSKINF) commands, as well as the Retrieve Directory Information (RTVDIRINF) and Print Directory Information (RTVDIRINF) commands to allow you to review where the disk storage is being consumed. Retrieve/print disk information is used to help you understand your storage usage for objects; retrieve/print directory information is used to help you understand your storage usage in the Integrated File System.

Tracking temporary storage can be a bit challenging. Since temporary storage is tracked on a job-level, you need to look at the job to see what it is doing. The general approach is to start with Work with System Status to review the overall disk storage consumption for the partition; “Current unprotect used” is the current amount of temporary storage being used and “Maximum unprotect” is the high-water mark, showing the highest value temporary storage has reached since the system was last IPLed. If you determine you may have an issue with temporary storage, then next step is to use the Work with System Activity (WRKSYSACT) command (beginning with the 6.1 release, WRKSYSACT is now part of the operating system and is available for everyone to use). Press the F11 key three times to get to the fourth view; it will have the storage allocated / deallocated columns were you can determine what jobs may be allocating more storage than they are releasing. Once you've identified the jobs that you want to look at further, you can then use the WRKJOB interface to determine what that job is doing.

There are a few important points to know about the storage allocated/deallocated columns on the WRKSYSACT panel:

• The storage allocated/deallocated columns include both temporary and permanent storage.
• It is possible for one job to be allocating the storage and a different job freeing it.
• Some types of temporary storage are not tracked by the system and you will not see it associated with any job.
• The storage allocated/deallocated values are deltas that are calculated from the last sample that was displayed.
• You can use the Sequence (SEQ) parameter on the WRKSTSACT command to sequence the display for storage allocated (*STGALC), storage deallocated (*STGDLC), or the storage net (*STGNET)
  Even if you use these parameters, you will still need to press F11 three times to see the storage allocated/deallocated columns.

Finally, for those of you that are knowledgeable about the collection services data, the QAPMJOBMI file also contains the storage allocated and deallocated metrics for a job. Like WRKSYSACT, these metrics include both temporary and permanent storage values.

For more information on this topic, see the redpaper titled “Best Practices for Managing IBM i jobs, spooled output.” 

 

 

IBM i 7.1: Jobs Exceeding Their CPU or Storage Limits are now Held

The Work Management team recently released PTF SI42845 for the 7.1 release that changes how IBM i manages jobs that exceed their CPU or storage limits.

The class object defines the processing attributes for a job. The routing entry in the subsystem description is used to determine which class object is used when a job is initiated. Two of these processing attributes within the class object are Maximum processing unit time (CPUTIME) and Maximum temporary storage allowed (MAXTMPSTG), which both have default values of *NOMAX.  Prior to this recent PTF, if values were entered for these parameters, the job would be ended if one of the limits was hit. For the maximum processing unit time, the job would be ended with CPC1218 (Job ended abnormally); for the maximum temporary storage allowed, the job would be ended with CPC1217 (Job ended abnormally). The cause for each of these messages tells you whether the job ended abnormally due to the maximum CPU time being consumed or the maximum temporary storage limit being exceeded.

The system can’t know if the job was actually near the completion of the work it had to do when it would end the job. It's possible that given a little more CPU time or temporary storage, the job would be able to run to completion. Because of the difficulty in predicting the upper CPU or temporary storage limits required by a job, along with the fact that the job would be ended when these limits were hit, many customers simply left these values at their default setting.

The above PTF that was recently released changes the behavior so that jobs are no longer ended when they have exceeded their maximum processing unit time or their maximum temporary storage limit. Rather, the jobs will be held. When a job is held by the system due to these conditions, a message will be sent to the QSYSOPR message queue:

•    CPI112D – Job held by the system, CPUTIME limit exceeded
•    CPI112E – Job held by the system, MAXTMPSTG limit exceeded

This change allows the system operator to determine whether the jobs should be ended or if they should be allowed to continue to run to completion. 

If you want the jobs to continue to run, you must change the limit that was hit and then use the Release Job (RLSJOB) command (you can’t release a job that’s above the limit). To allow these values to be changed, the Change Job command and the Change Job APIs have been enhanced.

The Change Job (CHGJOB) command has been enhanced with two new parameters:

•    Maximum CPU time (CPUTIME): The maximum CPU time parameter specifies the maximum processing unit time (in milliseconds) that the job can use. If the maximum time is exceeded, the job is held.
•    Maximum temporary storage (MAXTMPSTG): The maximum temporary storage parameter specifies the maximum amount of temporary auxiliary storage (in megabytes) that the job can use. This temporary storage is used for storage required by the program itself and by implicitly created internal system objects used to support the job. (It doesn’t include storage for objects in the QTEMP library.)  If the maximum temporary storage is exceeded, the job is held.

The Change Job (QWTCHGJB) API has been enhanced to support two new keys on the JOBC0100 and JOBC0200 formats:

•    Maximum processing unit time allowed, in milliseconds (1302)
•    Maximum temporary storage allowed, in megabytes (1305)

This PTF makes it easier for you to protect your system from the effects of a run-away job that either consumes more CPU than expected or uses more temporary storage than expected. By setting these limits larger than what any job should use, you can protect the system from the potentially negative affects of a run-away job. Because the job will be held rather than ended, the limits don’t need to be set perfectly. If either limit is hit, you can increase the limit with the change job command or API then release the job to allow it to continue to run. If the new upper limit is hit, the system will once again hold the job.

With the change introduced with this PTF, you should start to move away from the default *NOMAX values and set appropriate limits. Particularly with the temporary storage limit, you can prevent a system outage by setting an upper limit on the class object for the maximum temporary storage that a job can use (but be sure to keep that limit lower than the amount of storage available on the system). With the new behavior of the job being held when the limit is hit, you have the capability to assess and determine the best action for the job.

I'd like to thank Dan Tarara from the IBM i work management development team for his assistance in writing this blog article.

PTFs for Additional Work-Management Controls

The work-management team recently made available some changes with PTFs to help control the behavior of job creation for cases when a program may get in a loop submitting jobs. The change made with the PTF is to slow down the creation of new jobs when the system is approaching the limit for the maximum number of jobs (QMAXJOB system value). 

QMAXJOB defines how many jobs can be on the system and includes all jobs: jobs submitted to a job queue, active jobs and ended jobs with spooled output. When the system reaches the limit specified by QMAXJOB, no more jobs can be started and the system can become inoperable. 

There’s a warning message--CPI1468 “System job tables nearing capacity”—that’s sent to QSYSOPR when the system is nearing the limit of the maximum number of jobs. Support was added to customize when these messages are sent with the following PTFs:
•    6.1 - SI30171
•    5.4 - SI29585
•    This same function exists in 7.1 without additional PTFs

By default, the CPI1468 is sent when the job tables are 90-percent full. In some situations, that’s too late and you don’t have time to recover from the condition. Create a data area named QMAXJOBPCT in QSYS with the desired percentage value at which you want the message to be set. The PTF cover letter has more detail.

The new PTFs that slow down the creation of new jobs when the system is approaching the QMAXJOB limit are:
•    7.1 – SI40568 
•    6.1 – SI40569
•    5.4 – SI40571

With these PTFs, the system adds delays to starting new jobs when the number of jobs is at 90 percent of the QMAXJOB setting. Override this percentage by creating a data named QWTSBSDLY in QSYS; for example, you could have the threshold be set to 80 percent. There are no delays if the number of jobs in the system remains below that percentage.

The delays start at a half second, but are increased as the system approaches the QMAXJOB limit. A single delay isn’t noticeable, but if you have a job that’s looping starting jobs, these delays can add up and offer some protection to keep the system operational as you approach the QMAXJOB limit. You can still run out of job-table entries, but it will take longer and you’ll have a little more time to react to the situation.

In 2008, we wrote a Redpaper, “Best Practices for Managing IBM I Jobs and Output (and a few other special tips)." This Redpaper covers general recommendations for improved management of jobs and output. Provided in that Redpaper is a sample CL program source for a Submit Job (SMBJOB) exit program. The submit job exit program is the best way to control the runaway submission of many jobs.

Another set of PTFs were provided that change the default behavior of the system to briefly display the end-of-job display when an interactive job ends. Previously, when an interactive job was ended, the end-of-job display would remain until the user pressed enter or 30 seconds had passed.  With these PTFs, this delay is removed and the interactive jobs are ended immediately without delay due to the end-of-job message; the message is still sent, but is only seen briefly. This change is helpful for ENDSBS, for example. Prior to the PTFs, ending an interactive subsystem would generally not end the jobs right away because the end-of-job display would be displayed for 30 seconds for those jobs where the user didn't press enter. With these PTFs, the jobs are ended right away.

These  PTFs are:
•    7.1 – SI40867
•    6.1 – SI40866 

The PTF cover letters containing documentation on how to create a data area named QWTEOJDSP in QSYS to get the old behavior back.

i Can … Use Subsystems With Many Devices

For those of you with larger environments where you may have hundreds to thousands of users on a single partition, you’re probably aware of the old recommendation to spread the allocation of those devices across multiple subsystems by using workstation entries or communications device entries. The IBM-documented recommendation was no more than 250-300 devices per subsystem. The communications-device issue has generally resolved itself since fewer shops are using SNA today, and TCP/IP communications support doesn’t use communications devices.

However, workstation devices are still widely used today; telnet sessions all use workstation devices.

The subsystem job is central to workstation-device management. It handles putting up the sign-on display, as well as error-recovery processing if the session ends unexpectedly. Prior to 5.4, subsystem jobs were single-threaded and could only process one device at a time. Thus, if you had a situation where many devices were affected at one time (a network outage, for example), the subsystem job could become a bottleneck for the device error recovery processing. As a circumvention for this issue, IBM made a general recommendation that no more than 250-300 devices be handled by a single subsystem. To implement this recommendation, you had to define multiple subsystem descriptions and set up the necessary workstation entries to spread the devices across those subsystems. We wrote the Interactive Subsystem Configuration experience report to describe in detail how to perform this configuration.

In the 5.4 release, the subsystem job architecture was changed to be multithreaded; now a single subsystem job could handle device-error processing for up to 20 devices at a time, in 20 different threads. This architecture change had benefits beyond the parallel processing for device error recovery; it also resulted in subsystem start-up time being faster.

I want to note, though, that the controlling subsystem job isn’t multithreaded. If you leave the controlling subsystem system value (QCTLSBSD) to the default of QBASE, your interactive users will run in the controlling subsystem and you won’t get the benefits of multithreaded subsystem jobs. If you have many interactive users, you should change the QCTLSBSD to QCTL (or equivalent).

Although multithread subsystem jobs alleviate the requirement to set up and manage multiple subsystems for interactive users, you may still want to consider doing this, although for different reasons. Multiple subsystems can make it easier for you to manage the users on your system and offer additional options for performance tuning.

So why write this blog about something so old? I had a coworker ask me just last week if we'd yet removed the restriction of 250-300 devices per subsystem. That took me by surprise, but since this was an internal design change that did not affect externals, we never documented this. We just removed that old recommendation from the Information Center. The only place I know where there was external documentation about this change was a small update in a support technote that described the circumvention stating that the recommendation is no longer needed for releases 5.4 and higher.

Finally, I'll share a bit of my history with subsystems and device-recovery processing. In 1995 or so, I was asked to investigate the SNA error-recovery processing in general. It was during this job assignment I learned of the tight integration between SNA communications and IBM i work management, and where I discovered the lack of scalability in subsystem jobs due to their single-threaded design. At that time we didn’t have support for threads; they were added to the operating system in the V4R2 release. We looked at various ways to address this scalability limit, but all of the ideas at the time were too expensive and risky to implement. So we published the recommended limit of 250-300 devices per subsystem as the circumvention and lived with the issue. In 2002, my work assignment changed and I joined the work-management team. While in that job, I had the opportunity to focus on our subsystem architecture. By this time, threads were old news and the recommendation of changing our subsystem architecture to be multithreaded was accepted and delivered in the 5.4 release.  It was an amazingly talented group of software engineers that worked on this project and the quality of their work was outstanding!

i Can … Display the Status of Your IOA Cache Batteries

I assume most of you are familiar with the I/O adapters that have cache batteries on them and the importance of monitoring the status of those batteries to ensure they are replaced before they fail. In case you’re not: the IOA has a cache to improve performance; the application doesn’t have to wait for the data to be written to the disk. The cache is powered by a battery in the IOA to ensure the cache data is preserved even if the IOA is powered off or reset. However, if the cache battery is failing, then the data is written directly to disk and the cache isn’t used, which can result in significant performance degradation. As such, you should know when the cache batteries are failing to get them replaced before it affects your application's performance. 

Various ways to display the status of your cache batteries exist, including one very new method that we’ve recently made available via PTFs. I'm only going to briefly summarize the existing methods of checking the battery status, since I really want to share the new information.

The system is aware when the cache batteries start to fail, and sends message CPPEA13 to the QSYSOPR message queue.  The first-level message text for this message is somewhat vague – “Contact your hardware service provider.” However, in the recovery section of the message it says you should have the cache battery replaced. When this message is sent to QSYSOPR, you’ll also have a problem created in the problem log, and you can use WRKPRB to display the problem details.

You can use the Work with Disk Status (WRKDSKSTS) command to determine if you have cache batteries requiring attention. Press F11 to get to the second panel; if the status for the disk unit is DEGRADED, it’s possible the issue is due to failing cache batteries. If your drives are mirrored, however, they’ll always show ACTIVE status and you must use one of the alternative ways to check the cache battery status.

There are a two different ways you can view your cache battery status using System Service Tools (STRSST): 

1. You can use the Hardware Service Manager to work with resources containing cache battery packs, which will show you the resources that have cache batteries, and you can then display the battery information.

2. You can use the Display/Alter/Dump service function to invoke the batteryinfo Advanced Analysis feature to display the status of your cache batteries.


And now for the news: some recent PTFs (one for each supported release) now provide a simple program that you can call that to display the cache battery status.  They are:

7.1 - SI40406
6.1 - SI40404
5.4 - SI40403

To display the cache battery information, simply run the following:

CALL QSYS/QSMBTTCC

This will create a spool file containing the cache battery information for the IBM i partition where the function is run. To use this function you must have *SERVICE special authority, or be authorized to the Service Trace function. The Change Function Usage (CHGFCNUSG) command, with a function ID of QIBM_SERVICE_TRACE, can also be used to change the list of users that are allowed to perform this operation.

i Can … Remove Disk Units Without an IPL at 7.1

It's tough to make the new release highlights list. A valuable capability that will help reduce planned outages that didn't make the cut for IBM i 7.1 announcement highlights is the capability to remove disk units from the partition without requiring an IPL or going into restricted state. While it’s documented in the 7.1 Information Center, you most likely wouldn’t find it if you didn't know to go look for it. 

Starting with 7.1, you have capability to concurrently remove disk units in the system ASP, user ASPs,   and on various independent ASPs. Prior to 7.1, you had to manual IPL to DST in order to remove disk units in the system ASP or a user ASP.  If you wanted to remove units from an independent ASP, the independent ASP had to be varied off.

Just think of the possibilities! If you want to upgrade your disk technology to the latest and greatest, you can now do it with only a single outage to migrate the load source unit for a single ASP system.  For user ASPs or independent ASPs, you can achieve a complete upgrade of all disk units with no outage at all. 

The capability is access via System Service Tools (SST).  You'll notice a change on the Work with Disk Configuration panel. Click the following link to view panel: Download ICan Work with Disk Config panel 6.1.10

After selecting option 12, you will see a new panel. Click the following link to view panel: Download ICan Work with Removing Units From Configuration 6.1.10

As you can see from the options presented, not only can you start the remove process but you can pause the remove and resume one that’s paused. You can cancel the remove either with or without a balance.  You can also use either the option provided on the service tools panel or WRKDSKSTS to see the status of the remove operation.

Planning before beginning a remove of units will assure success with low business impact. Most important is to ensure enough free space on units that will remain to absorb the GBs from the disks you are removing. If the ASP threshold is exceeded while the remove process is running, the remove will be paused. 

Other items to consider while planning:
•    IBM i 7.1 is required.
•    Any server or blade hardware that supports IBM i 7.1 can be utilized.
•    If a remove is in progress when the system is powered down, ended abnormally, or an IASP is varied off, the remove will be paused. The remove must be restarted by choosing the resume option.
•    The load source unit cannot be removed from the system ASP.
•    Not all drives can be removed from an ASP. At least one must remain.
•    Moving drives between ASPs is not supported.The units must be removed from one ASP then added to the other ASP.
•    All units selected for removal must belong to a single ASP.
•    Only one ASP can be operated on at a time. Other ASPs removes may be paused but only one can be running.
•    No changes have been made to the remove processing that occurs during a manual IPL to DST.  A remove started at DST must complete before the IPL can continue.

Additional information about the remove process and other enhancements that increase your system's availability can be found online.

I'd like to thank Sue Baker for writing this blog article. Sue is a Certified Consulting IT Specialist working for IBM's Americas Advanced Technical Skills organization. Sue is currently focused on Power Systems hardware performance in the arenas of processor, memory, disk subsystems, and I/O. Sue consults with customers in the challenging arena of managing the operating environment to provide highly available, highly performing, and highly resilient Power Systems infrastructure.