Health Indicators in the Performance Data Investigator

The Performance Data Investigator has a wealth of capabilities, some of which I have written about in the past. This week, I want to review the Health Indicators.

Health Indicators are a set of graphs (perspectives) found within the Investigate Data task in the Navigator for i Web console. The health indicators are based upon Collection Services data and are included with the base operating system (you do not need to purchase or install any additional licensed programs to use the Health Indicators). Health Indicators give you an overall view of the performance of your partition and can be used to easily determine whether you should investigate the performance of your partition further.

As you can see from the screen capture above, there are various different types of Health Indicators. The System Resources Health Indicators provide you an overview of high-level performance metrics for the partition (CPU, memory, disk, etc.). The other types of health indicators give you a bit more detail. Below is an example of the System Resoruces Health Indicators:

It’s important that you understand how to interpret the graph that’s displayed. For each metric that’s displayed, the colors (green, yellow, and red) represent the percentage of intervals within the collection that had values that were within that threshold level.

It’s easiest to explain by referring to the example above. If we look at the CPU metric, we will see that around 70% of the intervals, CPU metrics were healthy (green). About 10% of the intervals, we have values that were in the warning level (yellow), and 20% of the intervals, we had values that were above the action level (red). This graph does not tell us when the warning and action thresholds were hit – only that they occurred in some percentage of the intevals within the collection.

We can drill into the CPU health indicators to see more detail:

From this chart we can now see that we have concerns about 5% of the time regarding the partition CPU Utilization, but the major issue is Jobs CPU Queuing, which was happening around 80% of the time during the collection. To better understand what’s happening, we can now drill into the CPU Utilization and Waits overview to look at the Collection Services data over time, to see when these situations were occurring:

In the resulting chart, you can now see when the CPU Queuing time occurred and when the partition CPU utilization exceeded the Action threshold (circled in red).

These threshold values have defaults supplied by IBM, but you can modify them by taking the Define Health Indicators from the action drop-down:

The work management System Status GUI provides links to the Health Indicators. The General tab will provide a link to the System Resources Health Indicators, as shown below.

If you select other tabs from System Status, you will find links to other types of health indicators: the Processors tab includes a link to the CPU Health Indicators, the Memory tab includes a link to the Memory Health Indicators, and the Disk Space tab includes a link to the Disk Health Indicators. These buttons allow you to go from the work management system status information to view the Health Indicators based upon the current Collection Services data.

As previously mentioned, the Health Indicators are based upon Collection Services data; you can view the Health Indicators for the most recent collection (near real time information – i.e., how is the system performing now), as well as in the past for any collections you still have available.

In summary, you can use the Health Indicators to get an overview of the performance of your IBM i partition; if you see areas of concern, you can then use the Performance Data Investigator to drill into the performance data details to determine what issues may be occurring on your partition.

 

 

IBM i Performance Analysis

If you are a regular reader of this blog, you no doubt have come to realize how very fortunate we are on IBM i in the realm of performance tools and performance management. From the seamless instrumentation of metrics, to the various performance data collectors (all included in the operating system), to the powerful graphical tools that are available to help analyze the data. Couple that with the benefit of the patented wait accounting technology, and it's clear that we have an impressive end-to-end suite of technologies at our disposal to monitor, manage and solve performance problems.

Now on to the intangible – the methodology. Even with really great tooling, analyzing the performance of a partition can be a complex task. The amount of data can be overwhelming. It can be hard to know where to start, what tool(s) to use, and whether the data is indicative of a problem or not. Does this sound familiar?

Let’s imagine the following scenario. Every day this week, the users have started to complain about their response times becoming unreasonable around 3:00 p.m., and the slowdown usually last for an hour. A typical methodology to use in the scenario would be to use leverage the real-time tools (Active Jobs, Disk Status, System Status, etc.) to gain initial clues on what system resources are being consumed (and a fleeting look at by who) during the particular troublesome timeframe. The next step would be to use the capabilities of Collection Services to determine in further detail who is using resources and what resources are being consumed. In addition to the who/what is being used, Collection Services is also used to find bottlenecks such as what resources jobs are waiting for (e.g., CPU, Disk I/O), and types of contention time if present. Sometimes, the problem can be solved with just this information. Perhaps during this timeframe we see a lot of CPU queuing time occurring. So next, we look at who is using CPU, and because we kept a baseline (you are keeping a baseline, right?), we look for changes. What we notice is that a new set of reports for management are now being run at 3 p.m. every day, instead of during off-shift hours, pushing us over the “knee of the curve” for CPU utilization. In this case, it may be as simple as rescheduling these reports. However, unfortunately, in many cases it is not so simple and we must continue the investigation. The next step then would be to focus our attention on Job Watcher data. If not already being collected, we begin to collect it. Job Watcher will give us an even more granular view of where time is being spent both at a system level and at a job level. It will give us additional clues on wait objects, call stacks and SQL being run. If the problem is not solved using Job Watcher, but we can establish it is I/O related, we may next turn to Disk Watcher. If we need to get down into the application and program level in order to completely solve the issue, we will use Performance Explorer.

Now, back to my original question “Does this sound familiar?”…..

Because we know that many of you nodded your head “yes”, the Lab Services IBM i Performance team was inspired to build a three-day workshop “IBM i Performance Analysis Workshop” to help teach you some of the techniques we use to analyze data and solve performance problems. This workshop is not focused around how to use the tools, but rather the methodology used. We will cover the strengths of the collector tools and when to use them, as well as core methodology and analysis techniques for CPU, Memory, I/O and various waits. Our goal is to aid you in building a foundation of performance methodology you can apply in your environment. If you are interested in more details, including enrolling in a public workshop, please see this link:

http://www-03.ibm.com/systems/power/software/i/support/workshops/performance-analysis.html

We also do custom on-site workshops as a Lab Services offering. If you are interested, please contact Mike Gordon at [email protected].

I’d like to thank Stacy Benfield for writing this blog. Stacy is a member of the Lab Services Performance team and was actively involved in the creation of this workshop. Prior to joining the Lab Services performance team, Stacy was part of the IBM i development team, working on performance tools, where she was the team leader of the Performance Data Investigator.

 

 

IBM Power Virtualization Performance (PowerVP)

This week’s blog was written by Kent Bruinsma. Kent is part of the IBM i development team with extensive experience in TCP/IP and SNA communications support. Kent is now part of the IBM i performance tools team and worked on the PowerVP product development, including the support to enable it on IBM i.

Do you have an IBM POWER7 server with virtualized workloads? If so, you probably have a need to monitor and manage the performance of these virtual workloads. IBM provides many performance tools at the operating system level that do a great job monitoring LPAR performance, but in order to see what is happening on the system level and how each partition is working in relation to the other partitions, you had to put together a jigsaw puzzle of information gathered from each partition. What is missing is a single view and collection of performance information at the system level. PowerVP fills this gap by providing a systemwide view of all of the LPARs on your POWER7 system with utilization information for the partitions. It also provides utilization information for the physical hardware that can be used to help you understand and manage the partitions and workloads on your system.

The PowerVP GUI illustrates Power Systems hardware topology in conjunction with resource utilization metrics and LPAR physical placement data to help you better understand the system. These resources include nodes, processor modules, chips, cores, Power bus links, memory controller links and GX I/O links. These resource utilizations are portrayed using a colorized heat technique. These colors and thresholds can be customized to suit a specific client's performance requirements in a meaningful way. The PowerVP GUI provides a real-time view of your system’s performance at it is occurring, sampling performance information as frequently as every second.

Product Features

The PowerVP GUI is your window to your POWER7 system performance.

Here is an example screen of the PowerVP GUI showing the partition CPU utilization on the top of the panel and showing the core utilization and Power bus utilization in the lower section. The mapping of physical cores to LPARs is represented using different colors (i.e., the purple partition has five dedicated cores outlined with purple in the lower part of the screen):

On the right side of the window, you will see your system information, including a histogram graph of the CPU utilization of the whole system. The “Playback” section gives you DVD type capabilities to record the PowerVP performance that can then be “replayed” later using the PowerVP GUI.

The PowerVP GUI also can be used to illustrate resource utilization and application efficiency at the partition level. The partition “drill down” panel provides resource utilization of the disk and Ethernet along with some CPI (cycles per instruction) analysis that can be used to gauge application efficiency and give you a picture of the memory affinity of your applications.

Here is an example of a partition drill down:

Analyzing memory affinity is a more advanced technical topic, which would lead you to other resources, including www.ibm.com/systems/power/software/aix/whitepapers/perf_faq.html or

www.ibm.com/systems/power/software/i/management/performance/resources.html.

If you click on the Disk Transfer Rate column or the Total Ethernet column, the lower half of the screen will show these metrics broken down for each individual disk or Ethernet adaptor on your system. The Detailed view: LSU CPI shown above is displayed by default and also when you click on the LSU CPI column in the upper half.

There is also a view with PowerVP that shows each partitions contribution to the total CPU utilization of your system. Here is an example of it:

PowerVP Use Cases

The potential use cases for PowerVP include but are not limited to the following examples:

You can take a benchmark snapshot of your system so that when you make changes, for example, using Dynamic Platform Optimizer or DPO, you can use PowerVP to compare these changes to your previous benchmark, providing you with the ability to confirm your changes (either a positive or negative or no impact).

You can use PowerVP when you add additional workloads, to see the impact a new workload is making to your existing workloads.

PowerVP can be used as a simple health check of your POWER system giving you a quick view of the current state of your system.

PowerVP can be used as a monitor and customized to show particular colors to alert operators to potential problem situations.

PowerVP can be used as your starting point for diagnosing performance problems on your system. You can view the current resource utilizations for all partitions to determine which partitions may be causing you problems so that you know where to look further.

A New View

PowerVP was created to compliment the current suite of performance tools in your toolbox. PowerVP focuses on new views that typically are not available with the OS-based performance tools. PowerVP provides you with a real time systemwide view of your systems performance. For more information on PowerVP, see http://www-03.ibm.com/systems/power/software/performance/index.html.

 

 

View Memory Pools and Faulting With the Performance Data Investigator

Recently I wrote the blog Work with System Status - the Navigator Way, which described how, using Navigator for i, you can make some simple customizations to get the Active Memory Pools view to show you additional information – pool sizes, fault rates, etc., and mentioned that the work management tasks link to the performance tasks.

This week, I want to expand upon the performance tasks side of this – and share with you some relatively recent enhancements in the Performance Data Investigator (PDI) that provide graphs of your memory information.

For the features I review in this blog article, the underlying data is provided by Collection Services, which you should be running all the time. The capability to view Collection Services data with the PDI is included with the operating system, so all of you have the capability to view these charts. You just need to make sure you have the latest Navigator for i PTFs.

Using the Performance Data Investigator, you can see memory and faulting information. It’s easiest to explain this with a few examples.

Once you are in the Investigate Data task, expand Collection Services, and you will find several folders – the ones of interest for this discussion are the Memory folder and the Page Faults folder. Each contains a set of perspectives (charts). The Page Faults perspectives have been there for a long time. The Memory perspectives are relatively new.

If you expand the Memory folder, you will discover the following perspectives:

Hmmm … this sure looks a lot like what we see on WRKSYSSTS, doesn’t it? It is … and it can be a bit better, particularly if you want to see the information over time. With WRKSYSSTS, you see the real-time values; if you want to take note of how they are changing, you either need to take screen captures or write down the values. But Collection Services collects this data on an ongoing basis and PDI allows you to visualize that data so you can see what has been happening over time. That might be very valuable in understanding what is happening on your partition. (Note: the granularity of what you see with PDI and Collection Services data will depend upon the collection interval you have configured.)

Below is an example view of the Memory Pools Sizes and Fault Rates perspective; on my partition, the performance adjuster (QPFRADJ) system value is on.

From this perspective, there are several drill-down options. One option is to drill down to “Page Faults By Job or Task”, where you identify what jobs were faulting. From this perspective, you can then drill into the page faults for one job to see that specific job’s behavior.

Below is another example screen capture, this one being “Memory Pool Activity Levels”. In this example, a workload started to ramp up around midnight and the work continued to increase and we can see this represented in our activity levels. (This was a stress-test workload done in the Rochester lab.) Performance adjuster was on.

If you expand the Page Faults folder, you will find more perspectives for viewing faulting data:

These are just a few simple examples of some of the perspectives you can view. Take the time to look at your Collection Services data with the Performance Data Investigator – you’ll likely learn a lot about what is happening on your system.

If you are not familiar with the Performance Data Investigator, the article I referenced earlier is a good place to start. There are also many additional resources on the IBM i performance tools page Resources page on developerWorks.

To learn about the new features that have been added to PDI, refer to the Performance on the Web documentation on developerWorks. The memory perspectives were part of the Spring 2013 update.

 

 

 

Work With System Status – the Navigator Way

I’m sure many (most?) of you still use Work with System Status (WRKSYSSTS) as the way to review your overall system status, including memory pools and faulting on IBM i. After all, that first view gives you a lot of useful information…CPU utilization, jobs in the system, storage used, and of course, faulting information. My screen capture below is uninteresting.

What if I suggest another way to look at this same information? Where you can customize what you want to see? You just need to use the GUI, in particular, the Navigator Web console.

(Note: you can also do this in System i Navigator, but as you get to the end of this blog, you’ll learn of options that are only in the Navigator for i Web console).

If you have been using WRKSYSSTS and want to explore the GUI, you’ll find Systemà System Status and may assume that will give you the WRKSYSSTS information on the GUI. Perhaps you tried this with Navigator before and didn’t find what you were looking for? The information on the GUI via System Status is much of what you find on the top half of the WRKSYSSTS display.

Since System Status doesn’t give you the faulting information, you need to go to the Active Memory Pools view.

But wait! When you open that view, you don’t see faulting information there either!

There’s a trick! You simply need to customize the columns that are displayed – there are many places in the GUI where you can get at more information by customizing columns. Take the Actions drop down and go to Columns…

Add in the columns you want and adjust the order to suit your preferences. The changes you make are saved and you’ll see that customized version from now on.

Here’s a screen capture of my version of Active Memory Pools – I have all the metrics I want to see in one table. Again, my system is uninteresting.

Another hint: if you select a specific memory pool (one that has a lot of faulting, perhaps), you can link over to view the jobs and subsystems associated with that memory pool.

And here’s the treat! There’s another big advantage when you use the Navigator GUI –integration between the work management tasks and the performance tasks, which I wrote about in the blog Work Management Integration with Performance Tasks.

In the case of Active Memory Pools, you can go directly to the Memory Pools Health Indicators in the Performance Data Investigator.

The Memory Pools Health Indicators can give you an indication of whether your faulting rates may be too high.

From the Memory Pools Health Indicators, you can then drill down into more detailed information regarding your memory pool sizes and fault rates using Collection Services data. And I’ll write more about that next week!

 

 

Enhanced SMT Controls for POWER7

Simultaneous Multi-Threading (SMT) allows sharing of processor facilities to run two or more threads of execution at the same time on a single processing core. On POWER7, each processing core supports up to 4 threads of execution. SMT generally increases the performance capacity of a system and can improve the responsiveness of multithreaded applications. Running multiple instruction streams at the same time does not, however, improve the performance of any given task.

The Processor Multitasking system value (QPRCMLTTSK) is used to control how SMT is used on the partition. The system value has 3 options:

• Off (0) – Each processor supports the processing of only one task at any moment.
• On (1) – Processors can concurrently process multiple tasks.
• System controlled (2) – Processors can concurrently process multiple tasks. The system-controlled state also enables the operating system to occasionally limit a processor to processing one task at most.

In the past, changes to this system value required an IPL to take effect. But now with enhanced SMT controls in IBM i 7.1 with TR5, you can switch between SMT modes immediately without an IPL. We have also introduced a new set of APIs to allow for additional controls to limit the number of hardware threads per processor. The Change Processor Multitasking Information (QWCCHGPR) API can be used to change the maximum number of secondary hardware threads per processor and the Retrieve Processor Multitasking Information (QWCRTVPR) API can be used to view the current setting on the partition.

In most cases, the system value should be set to 2 (system controlled) and there should be no need to change it. But if a situation arises in which IBM support determines your workload could benefit from limiting the number of processing threads, the system value along with the additional API controls can be used to make an immediate change.

For more information, see the following links in the IBM i Information Center and IBM developerWorks:

QPRCMLTTSK system value

Change Processor Multitasking Information (QWCCHGPR) API

Retrieve Processor Multitasking Information (QWCRTVPR) API

“Enhanced Administrative SMT controls for POWER7” section in the IBM i Functional Enhancements article. 

 

I’d like to thank Angela Newton for writing this blog article. Angela is the team leader of the IBM i work management team and is the IBM i expert for the development of APIs within the operating system. Thanks, Angela!

 

IBM i Wait Accounting

A long time ago I wrote about IBM i wait accounting in i Can …. Tell You Why You’re Waiting. That blog is still accurate and, for those of you interested in analyzing performance, very important; wait accounting is a very powerful tool for performance analysis.

One of my long-standing “to-do’s” was to write a more in-depth article about wait accounting and also include a description of all the wait buckets, written in a way that is meaningful to more people (i.e., you don’t have to be a developer of the IBM i operating system to understand what the wait buckets mean!). That article is finally done.

Very recently, the “IBM i Wait Accounting” article was published on the IBM i developerWorks site. This article explains the wait accounting technology in a little more detail and provides some information on how to get started analyzing the wait data that is available in Collection Services or Job Watcher by using the Performance Data Investigator.

At the end of the article, I’ve included a description of each of the wait buckets. Hopefully this is a helpful resource for those of you who are learning about performance data and analysis on i.

 

Viewing Job Level SQL Metrics with the Performance Data Investigator

Previously I wrote about Job Level SQL Metrics in Collection Services. In this week’s blog, I will review the support added to the Performance Data Investigator (PDI) to allow you to view and analyze these job-level SQL metrics.

First, you must have the latest Navigator for i PTFs installed as reviewed in the blog Navigator for i Enhancements. You will also want to verify that you have the PTFs installed and activated that were noted in the Job Level SQL Metrics in Collection Services blog.

The GUI for the new SQL metrics is contained in a content package named Database. This content package will only show up under the list of perspectives in the Performance Data Investigator if you have the Performance Tools licensed program product (5770-PT1) installed; you just need the manager option. Put another way, the enablement for the database graphical interface in the Performance Data Investigator is shipped with PT1.

Once you have the latest Navigator PTFs installed (along with the performance tools and database group PTFs that I reviewed previously), you will discover you now have some new perspectives available in the Performance Data Investigator. In particular, you will notice a new Database folder that includes Database I/O perspectives and SQL Performance Data perspectives. The following screen capture shows this:

You need to drill down into the Database I/O and subsequent folders to find the graphs and charts of data you can now view on Physical Database I/O metrics – as you can see from the following screen capture, you have several views of these database metrics:

If you have not followed the activation instructions required for the Collection Services PTF I previously wrote about, you will get an error if you try to view one of these physical database I/O charts. This is because the underlying data is not collected. Refer to SI47594 PTF Special Instructions for details.

An example of the Physical Database I/O Overview chart is below. You can see the SQL-related physical I/O metrics are summarized by time for the selected collection.

There are several different ways you can view the physical database I/O – by job, thread, user profile, current user profile, subsystem, or server type – to give you various ways to view and analyze the metrics.

You will note that the Performance Data Investigator now also supports viewing data from the SQL plan cache and the SQL performance monitor. I don’t have time in this blog to go into details on using PDI to look at this data, but for those of you who are interested in database performance, it’s worth getting current on your PTFs and checking out these new features.

The Performance Data Investigator development team has created documentation on developerWorks that summarizes the latest enhancements that have been made. There is an article on the new Database content package that you can look at for more information.

Finally, there are some recent changes that were not discussed in the prior blog regarding the SQL metrics. My previous blog stated that this was only supported on the 7.1 release, but the database metrics are now also available on the 6.1 release! Please read the Spring 2013 documentation (which is a subset of the Performance on the web documentation) on the IBM i developerWorks site to get the details regarding the support added to 6.1 and the required PTFs. If you read that Web page, you’ll discover that I have even more to blog about in coming weeks!

 

 

Anticipating the Future

Up to 1,000 partitions, capped vs. uncapped, processor pools, dynamic processor allocation, Capacity on Demand, Live Partition Mobility, multiple OSes, etc., etc. All of these are extremely valuable capabilities built into Power Systems and IBM i that help you get the most out of your investment. But does tracking their impact on your performance and utilization make it challenging to “stay on top” of what you should be planning for next? And then what do you do when a new processor technology is announced and you try to anticipate how it would impact your environment?

Performance Management (PM) for Power Systems is designed to help you with this. PM is an integrated data collector within IBM i that automatically and continuously collects utilization and performance data and subsequently makes it available for your use. The collected data is securely sent to IBM as part of the “call home” function built into your system as part of Electronic Service Agent or other “call home” facilities (e.g., the Service and Support Manager in IBM Systems Director with an HMC or FSM installed). The data is securely stored and graphed by IBM on a rolling 25-month basis.

The value? IBM constantly trends the data to identify growth patterns across many facets of the system. That growth is compared to available capacity at the partition, processor pool and total system level. In most cases, potential problems are flagged in advance...so that you have time to plan for what is needed before a crisis occurs. You can merge your collected data, with a few mouse clicks, with the IBM Systems Workload Estimator to “size” what additional resources to plan for next (e.g., a capacity on demand processor? POWER7+ ?).

The following figure is an example of how PM can show you a total system view, including available CoD processors.

Many of these functions (including the sizing function) are available for no additional charge as part of IBM i. (Support is also provided for AIX and Linux.) The more detailed graphs are available as part of an IBM Global Services offering. You can learn more about PM and how the detailed reports are packaged in your country at the IBM PM for Power Systems website.

I’d like to thank Van Sammons for writing this blog article. Van is part of the Performance Management for Power Systems team in IBM Global Technology Services. Thanks, Van!

Job Level SQL Metrics in Collection Services

Late last year, IBM enabled new support to collect SQL metrics in Collection Services. I’ll be covering this new capability in two blog articles:

1. This blog will review the new metrics that are now collected and describe what you need to do to enable the collection of these additional metrics.
2. A subsequent blog will review the support added to the Performance Data Investigator to allow you to view and analyze these metrics.

Collection Services collects a wealth of information, but metrics related to SQL operations were limited.  The tools used for DB2 and SQL performance analysis are separate from the system performance tools; interfaces such as the DB2 Health Center, SQL Plan Cache and Index Advisor, among others, are what you need to use to understand database performance issues.

The most recent PTFs augment the tools already available for SQL performance analysis and there are now SQL metrics collected at the job level. This allows you to begin your performance analysis with Collection Services and determine if SQL operations are a contributing factor. You will probably still use the DB2-specific tools for digging into the details, but you now have insight into the SQL operations from Collection Services data.

This enhancement is only available on IBM i 7.1 (i.e., it is not available on 6.1) and requires that you have installed the latest DB2 Group PTF SF99701, as well as SI47594, which provides the Collection Services enablement. It’s also a good idea to ensure that you have the latest Performance Tools Group PTF, SF99145, installed as well.

Just installing the required PTFs is not sufficient. There are activation instructions you must take to enable the collection of these metrics documented in the SI47594 PTF cover letter. The instructions are rather lengthy but not difficult to follow. The Job Level SQL Metrics article on developerWorks also documents these activation instructions.

The reason for these special instructions has to do with the fact that the file, QAPMJOBMI, needs to be updated to a new level to support these additional metrics.  Specifically, an updated template file for QAPMJOBMI is shipped in QSYS with the additional fields. This template file will only be used when creating a new QAPMJOBMI file in libraries where it doesn't already exist. Existing collection libraries (for example, QPFRDATA) that contain a prior version of QAPMJOBMI will not be affected by the PTF.

Once you have the required PTFs installed and completed the necessary activation instructions, you will have SQL metrics in your job-level Collection Services data. The Performance Data Investigator can then be used to view and analyze these metrics. As I mentioned earlier, I’ll write more about the PDI support in a future blog. To be prepared to use this new capability, review the blog Navigator for i Enhancements and install the PTFs as documented in that blog.

The following are the new metrics now available for collection:

New Collection Services metrics for SQL:

The following SQL performance metrics are collected as part of the *JOBMI category.

1. SQL clock time (total time in SQ and below) per thread (microseconds)
   The amount of clock time (in microseconds) this thread has spent performing work done on behalf of an SQL operation.
2. SQL unscaled CPU per thread (microseconds)
   Thread unscaled SQL CPU time used. The amount of unscaled processor time (in microseconds) this thread has used performing work done on behalf of an SQL operation.
3. SQL scaled CPU per thread (microseconds)
   Thread scaled SQL CPU time used. The amount of scaled processor time (in microseconds) this thread has used performing work done on behalf of an SQL operation.
4. SQL synchronous database reads per thread
   Total number of physical synchronous database read operations done on behalf of an SQL operation.
5. SQL synchronous non-database reads per thread
   Total number of physical synchronous nondatabase read operations done on behalf of an SQL operation.
6. SQL synchronous database writes per thread
   Total number of physical synchronous database write operations done on behalf of an SQL operation.
7. SQL synchronous non-database writes per thread
   Total number of physical synchronous nondatabase write operations done on behalf of an SQL operation.
8. SQL asynchronous database reads per thread
   Total number of physical asynchronous database read operations done on behalf of an SQL operation.
9. SQL asynchronous non-database reads per thread
   Total number of physical asynchronous nondatabase read operations done on behalf of an SQL operation.
10. SQL asynchronous database writes per thread
    Total number of physical asynchronous database write operations done on behalf of an SQL operation.
11. SQL asynchronous non-database writes per thread
    Total number of physical asynchronous nondatabase write operations done on behalf of an SQL operation.
12. Number of high Level SQL Statement count per thread
    The number of high level SQL statements that executed during the Collection Services time interval. This count includes only initial invocation, independent SQL statements. It does not include dependent SQL statements invoked from within another SQL statement. This count also includes initial invocation, independent SQL statements which failed to execute successfully.