RPG Speed Dating

Remember Y2K? It is somewhat mind-blowing to realize just how many years have elapsed since then. We were both working for IBM at the time and were heavily involved in IBM's efforts to provide tools and education to remediate the problem. In fact teaching IBMers and ISVs about IBM's Bypass2000 product took both of us around the world including our first (and to date only) visit to China. It was an interesting time.

One thing that surprised us as we discussed remediation plans with various IBM users was how few intended to take advantage of what we came to know as "real date fields" as part of their strategy. By "real" we mean DDS type L (SQL type DATE) as well as type D fields in RPG. When we questioned the rationale, two major reasons were given:

1. It took up more disk space than the four bytes used to hold a fake date in packed format. In fact this is not true. On disk a real date is held as a 4 byte integer. We just don't get to see it in that format in our programs.

2. A performance penalty was incurred every time a record was read to convert the date from the stored integer to the 8 or 10 character representation that we see in our programs. This of course is true, although much of the time bad RPG programming practice caused the performance hit to be far worse than it needed to be. Our feeling then was that for most shops the advantages far outweighed the performance difference in most cases. We still feel that way.

In recent years we have seen more and more people begin to take advantage of real dates. Many factors are at play in this increased uptake. The increase in the use of SQL and DDL-defined tables is part of the picture. The massive performance improvements in processor and disk speed is another.

But things can still get better, and the good folks on the RPG compiler team decided to add their own performance boost to date- and time-handling. This support actually came about some time ago, but since it was made available via PTF vs. a new release boundary, many people missed hearing about this when it occurred.

This support is available via PTF for both V7.1 (PTF SI43157) and V6.1 (PTF SI43156). It's a new H spec (or Ctl-Opt operation) keyword. The VALIDATE(*NODATETIME) option can speed up programs that do significant amounts of date- and time-handling. It achieves the performance boost by bypassing something you probably didn't even know it was doing - validating _every_ date field _every_ time it is used. Yup, _every_ time it is used.

With this option in play, under some circumstances the compiler will just treat the date as a character field and assume that it contains a valid date and bypass the validation. What circumstances? The following are some examples of cases where validation will not be done. Other circumstances may be added to the list later.

• When one date field is moved to another with the same format and separator. This includes the moves that occur when the field is read from the database, assuming the same format externally and internally, which is not always the case, depending on how the I/O operation is done and/or what the default date format is in the program.
• When a comparison, search or sort operation is being made between *ISO or *JIS formatted dates (i.e., YYYY MM DD) and between times in any format other than *USA. Using this option can have significant performance benefit for programs that handle a lot of dates in these categories. However, remember that some risk is still involved since the compiler will not be validating all date and time values. These are the main points to remember to take the biggest advantage of this new support:
• Make sure your code handles dates and times appropriately and that you are confident of all date values--especially any that may be definted in data structures.
• Strive for date formats and separators that are consistent both in your external database definitions and internally in RPG. Use *ISO (or *JIS) as the format of your date fields wherever feasible to gain the most benefit for comparisons, searching and sorting. Avoid *USA for any time fields.
• Watch out for a DATFMT keyword on H specs (or Ctl-Opt operations). This will change the default format for the program and may inadvertently result in non-ISO formats. If you don't specify DATFMT, the default will be *ISO, which is one of the preferred formats.
• When date data will be moved/assigned in the program, use the same date and time formats and separators as much as feasible. Don't forget that this applies not only to Eval (or heaven forbid, MOVE operations) but also to parameters defined using VALUE or CONST or function return values.
• If your external date fields are not *ISO format, then ensure that those fields are defined in Data Structures so that they will retain their external formats. This way the movement to/from I and O specs by the compiler can use the faster "no validation" option.  

You can get many more details about this support from a notice on the RPG Cafe in DeveloperWorks. On that page, you'll see a PDF attachment with many details and helpful advice on how to make the most of this feature.

The compiler team intends to increase the number of cases where validation is bypassed in subsequent releases, so if you use this option make sure to watch for future announcements in the RPG Cafe that may affect your code.

CL Rides Again

Those of you who have followed our writings (ramblings?) for a while know that we have a kind of love-hate relationship with CL. The syntax, while effective for commands, just feels clunky and unnatural when used for an entire program. Back in the pre-RPG IV days we would begrudgingly call CL programs to do the stuff that we just couldn't do in RPG. But as RPG IV evolved, and the number of system APIs increased, there was less and less that we couldn't do directly in RPG. Combine that with the fact that CL effectively stagnated for many years and there was little to convince us otherwise, hard as CL aficionados such as Bruce Vining and Jim Sloan tried to convince us.

But things have changed. Under the stewardship of Guy Vig, CL has been given a new lease on life with enhancements in many areas. It started back in OS/400 V5R3 with additions to the types of variables that could be defined, new looping and control structures, increases in the number of files that could be declared and more. The enhancements continued in V5R4 with the addition of subroutines and enhancements to data definition that effectively provided a form of data structure (DS) definition. Since having to use string/substring operations to fake out DS definitions was one of our pet peeves in CL, this was a big step forward. IBM i 6.1 continued the forward movement with the addition of INCLUDE, the equivalent of RPG's /COPY. It also finally added a CLOSE command--now you can actually close a file that has been opened in the CL program and open another (or the same) file in the program without performing unnatural acts.

Now with the latest round of PTFs to 7.1 a number of our remaining frustrations have been addressed and we really have been forced to start to take more notice of this IBM i stalwart. To use this new CL functionality on a 7.1 system, PTF SI49061 must be loaded and applied*. Note that support is _not_ going to be PTF'd to the 6.1 or 5.4 releases, but you will be able to compile CL code that uses these new functions on your 7.1 system by specifying the appropriate target release (TGTRLS) value.

These latest enhancements provide some major upgrades to CL's string handling capabilities. Even better, because they are almost identical in name and operation to their RPG equivalents, there's nothing much new to learn. The new functions are:

%CHECK    - Check Characters
%CHECKR - Check Reverse
%SCAN    - Scan for Characters
%TRIM    - Trim Characters at Edges
%TRIML    - Trim Leading Characters
%TRIMR    - Trim Trailing Characters 

These new built-ins can be used wherever CL would permit the use of an expression of the same type. For example %CHECK can be used in an IF or WHEN command. %TRIM (and its variants) can be used directly in a character expression (e.g., CHGVAR VAR(&ADDRESS) VALUE(%TRIM(&STREETNAME) *BCAT %TRIM(&CITY)) ) or to set the value of any command parameter that permits expressions (i.e. ,the parameter is specified as EXPR(*YES) ). You can get more details on the %TRIMx functions here and on the %CHECKx and %Scan functions here.

We understand more enhancements are planned for CL in the future, but nobody will actually tell us what they are yet so we'll all have to wait. What is clear though is that the days of CL as a stagnant, unchanging language are over and that we really need to start paying more attention to it.

Many thanks to Guy and the CL team!
 
* Note: The %Trim functions were actually snuck in via PTF back in October 2012 but went largely unnoticed.

The Cost of Standing Still

We are constantly amazed (more like dismayed) at the number of shops we run into at conferences and during private classes who are still using V5R4. Since IBM has now announced that end of life for V5R4 support is in September of this year, we're wondering how that will affect these shops.

We suspect that many end to need an official end-of-life date to give them the impetus to move. So maybe some will finally get moving. After all, 6.1 has been available since 2008 and  7.1 has been around for more than two years. So we're not exactly talking about joining the bleeding edge!     

Speaking of 7.1, the other thing that has surprised us is the number of people on V5R4 who tell us they have plans to move to 6.1 soon. Why 6.1? If you've waited this long, surely one advantage of upgrading is that you can short-circuit the process by going directly to 7.1. Why move to 6.1 now just to get caught up to 2008 technology? Of course the great thing about moving to 7.1 is that you will be able to add new functionality by applying technology refreshes--something that has always required a release upgrade in the past.

We know many other reasons--besides inertia--keep shops at V5R4. We've heard of application providers who are charging high prices for versions that will run on 6.1 and later. In some cases this is happening even when the "upgrade" simply requires that a few modules need to be recompiled. And then of course a few applications are in use that can't move to 6.1 at all, because the company is now out of business and the code was compiled prior to 5.1 and has no observability.

Maybe other reasons also exist. We're interested in hearing about what's keeping shops from moving beyond V5R4 after all this time. Hopefully some of you will share your reasons with us in the comments. If you have recently upgraded or are planning to upgrade to 6.1, we're also curious what kept you from going directly to 7.1.

Obviously, there is effort and time involved in the upgrade process--perhaps more so in the move beyond V5R4 than from other releases. But every coin has two sides. There is a cost to staying on an old release as well.

Once IBM support goes away, you may also incur a cost related to your compliance with various standards, such as PCI. And there's the cost of not implementing new features and functions that require later releases.

We've written a lot about the new features in RPG in recent releases, including, but certainly not limited to, RPG Open Access, which can greatly simplify your RPG logic. Database has some great new features to offer as well as well as a long list of other functions supported in 6.1 and those found in 7.1.

So what's keeping you--or other shops you know about--at V5R4? Inquiring minds want to know.