RPG Speed Dating

Remember Y2K? It is somewhat mind-blowing to realize just how many years have elapsed since then. We were both working for IBM at the time and were heavily involved in IBM's efforts to provide tools and education to remediate the problem. In fact teaching IBMers and ISVs about IBM's Bypass2000 product took both of us around the world including our first (and to date only) visit to China. It was an interesting time.

One thing that surprised us as we discussed remediation plans with various IBM users was how few intended to take advantage of what we came to know as "real date fields" as part of their strategy. By "real" we mean DDS type L (SQL type DATE) as well as type D fields in RPG. When we questioned the rationale, two major reasons were given:

1. It took up more disk space than the four bytes used to hold a fake date in packed format. In fact this is not true. On disk a real date is held as a 4 byte integer. We just don't get to see it in that format in our programs.

2. A performance penalty was incurred every time a record was read to convert the date from the stored integer to the 8 or 10 character representation that we see in our programs. This of course is true, although much of the time bad RPG programming practice caused the performance hit to be far worse than it needed to be. Our feeling then was that for most shops the advantages far outweighed the performance difference in most cases. We still feel that way.

In recent years we have seen more and more people begin to take advantage of real dates. Many factors are at play in this increased uptake. The increase in the use of SQL and DDL-defined tables is part of the picture. The massive performance improvements in processor and disk speed is another.

But things can still get better, and the good folks on the RPG compiler team decided to add their own performance boost to date- and time-handling. This support actually came about some time ago, but since it was made available via PTF vs. a new release boundary, many people missed hearing about this when it occurred.

This support is available via PTF for both V7.1 (PTF SI43157) and V6.1 (PTF SI43156). It's a new H spec (or Ctl-Opt operation) keyword. The VALIDATE(*NODATETIME) option can speed up programs that do significant amounts of date- and time-handling. It achieves the performance boost by bypassing something you probably didn't even know it was doing - validating _every_ date field _every_ time it is used. Yup, _every_ time it is used.

With this option in play, under some circumstances the compiler will just treat the date as a character field and assume that it contains a valid date and bypass the validation. What circumstances? The following are some examples of cases where validation will not be done. Other circumstances may be added to the list later.

• When one date field is moved to another with the same format and separator. This includes the moves that occur when the field is read from the database, assuming the same format externally and internally, which is not always the case, depending on how the I/O operation is done and/or what the default date format is in the program.
• When a comparison, search or sort operation is being made between *ISO or *JIS formatted dates (i.e., YYYY MM DD) and between times in any format other than *USA. Using this option can have significant performance benefit for programs that handle a lot of dates in these categories. However, remember that some risk is still involved since the compiler will not be validating all date and time values. These are the main points to remember to take the biggest advantage of this new support:
• Make sure your code handles dates and times appropriately and that you are confident of all date values--especially any that may be definted in data structures.
• Strive for date formats and separators that are consistent both in your external database definitions and internally in RPG. Use *ISO (or *JIS) as the format of your date fields wherever feasible to gain the most benefit for comparisons, searching and sorting. Avoid *USA for any time fields.
• Watch out for a DATFMT keyword on H specs (or Ctl-Opt operations). This will change the default format for the program and may inadvertently result in non-ISO formats. If you don't specify DATFMT, the default will be *ISO, which is one of the preferred formats.
• When date data will be moved/assigned in the program, use the same date and time formats and separators as much as feasible. Don't forget that this applies not only to Eval (or heaven forbid, MOVE operations) but also to parameters defined using VALUE or CONST or function return values.
• If your external date fields are not *ISO format, then ensure that those fields are defined in Data Structures so that they will retain their external formats. This way the movement to/from I and O specs by the compiler can use the faster "no validation" option.  

You can get many more details about this support from a notice on the RPG Cafe in DeveloperWorks. On that page, you'll see a PDF attachment with many details and helpful advice on how to make the most of this feature.

The compiler team intends to increase the number of cases where validation is bypassed in subsequent releases, so if you use this option make sure to watch for future announcements in the RPG Cafe that may affect your code.

OCEAN Views

Last week we spoke at the OCEAN User Group's annual conference in Costa Mesa in (normally) sunny California. It was great to see that like several other IBM i events lately, their numbers were up from previous years at the main event. They also had excellent attendance at the dinner and keynote session, featuring Alison Butterill talking about modernization and at a pre-conference workshop on Ruby for IBM i.

This year the group also tried out the idea of running in-depth workshops including hands-on Labs on the Saturday after the main event. We taught an introductory RDi class, IBM's Dan Cruikshank taught SQL topics including IBM Data Studio and Larry Bolhuis taught systems management. We're told that there were approximately 100 attendees at the workshops, which means that 100 people gave up part of their weekend to learn--and that is really encouraging.

OCEAN is a great group and if you live in reach of the Orange County, California-area you should definitely be a member.

OCEAN works with a local technical university to host the main conference and the workshops (the keynote and dinner were held at a hotel to accommodate larger numbers in a single session). We've mentioned this idea of using colleges for conferences before. It really can be a practical solution for local user groups that have considered doing events but found hosting them in hotels expensive and inflexible. In fact it does seem to be a bit of a trend with TUG, our local Toronto user group, holding its TEC event at a local college. We have heard that the OMNI group is working on a similar idea for an upcoming event in the Chicago area.

Other Items

In last week's blog we mentioned punch cards in what reader Bob L. considered a derogatory fashion:

"Don't belittle the punched card. Its legacy lives on: after all, that is why you still have to remind those "young young" programmers that the code must start on or after column 8 and end at column 80!!"

We certainly didn't mean to belittle the punch card (except for maybe those silly little 96 column things) they were really useful for writing notes. Jon remembers that in his early days of presenting, notes on punch cards were a godsend because their very size forced you to keep to bullet points so you were never tempted to just read your presentation. These days he couldn't do that even if he could find a supply as he so rarely wears a jacket to keep them in.

Before we go wanted to thank those readers of our recent article on tools to help you trace program execution, "Tracing Our Steps," who pointed out alternative tools. We haven't had a chance to check them all out yet but we will do so and report back in the article comments and also here in the blog.

We actually get to spend the next two weeks at home, which will be a rare treat this summer. Who knows? Maybe one or two chores around the house might even get done!

Programming Highs and Tire Lows

Last week we experienced an interesting mix of high and low points.

The big high for us was teaching a class of Java programmers about the joys of RPG. We also spent a little time on RDi, but since they already used Rational Application Developer for their Java work that was mostly just to introduce them to the IBM i specific features.

Three things helped make this an enjoyable class to teach:

1. The students were all young--and we don't mean young in our terms (i.e., anyone less than 50) we mean young young.
2. They were all enthusiastic and asked the right questions, which clearly showed that they were not only understanding the language but also getting a feel for its power and simplicity. We'll be going back to complete their training later this year once the students have had a chance to absorb the initial lessons and we're looking forward to it.
3. This was our first opportunity to teach the new all-free-form RPG flavor of the language. While it was a lot more work to retrofit our teaching materials than we had anticipated (never thought about how many of the examples have F, D and P specs in them!), what a difference it makes when teaching RPG to Java (or C/C++/C# or PHP, etc) developers. Those pesky columns--even when just for data declarations--always caused so much consternation in past classes. This time, the only hitch was the occasional need to remind them that the code must start on or after column 8 and end at column 80.

This class has reaffirmed our belief that all too often, trying to replace your retiring RPG programmers with experienced RPGers is a waste of time. Not that there aren't good RPGers out there looking for work, but most are too expensive for companies are looking for junior to intermediate level staff. Either that or their "30 years of experience" is actually three years repeated 10 times over because they are so reluctant to learn new skills. Case in point--and one of the lows of the week--we had an email concerning a shop where the staff (we won't dignify them by using the terms "programmers", "developers" or "RPGers") are busily stripping out things like embedded SQL and prototyped program calls whenever they have to maintain a modern program because they can't be bothered to learn this "new-fangled stuff." They probably rue the passing of the tabulator and the punched card too. Sad, so sad. No wonder the shop is moving away from the IBM i.

The other high/low mixture of the week concerned our rental car. Very soon after we set off from the airport a low tire pressure warning appeared. We checked but the tire appeared to be fine. Given that rental car companies often do a lousy job of what would be the PDI (pre-delivery inspection) at your local car dealer we have encountered such false warnings many times before. We kept checking but despite the warning light the tire appeared fine. Until it wasn't. Suddenly it lost pressure so we took it to the local tire store to see if they could fix it. Nope. It had a side wall leak, which couldn't be plugged. No problem, they inflated the tire again and we called the rental car company. "Yes sir - we will happily exchange your car. Just drive to your nearest location and they will help you." Fine except only corporate locations would do and the nearest was a 90+ minute round trip away and it was already getting late in the day. It was a slow leak so hopefully it would last until the next evening when we could drive down and swap the car.

The next morning arrived sunny, hot and humid with the tire completely flat. Changing a tire while in work clothes and 95 degree weather had not been high on Jon's wish list for the day but. ...

Luckily that was the end of the lows. We drove the car back to the rental company and, to cut a long story short, the only replacement vehicle they had available was a Mercedes 350. Not a bad swap for a Chevy Impala! Only problem is that Jon is enjoying driving it so much that he wants to keep it. Sadly that is not on the cards but at least we get to use it for a few more days.

  

RCAC: Two Steps Forward ... One Back?

We were recently asked about the impact on existing RPG programs of the new Row and Column Access Control (RCAC) DB2 support in V7.2. For those who haven't yet heard about RCAC, the simplest explanation is that it allows you to control access to data by row (aka record) and by column (aka field). The column control masks specific data from people who don't need (or shouldn't be able) to see it. Think seeing masked data for salary, credit card numbers, etc. The row control simply omits access to the entire row for unauthorized users, much as a select/omit LF with specific user authorizations could do--only it's probably a bit easier to implement.

We haven't fully thought through some of the implications of row control yet. Clearly there is a difference between "These are all the rows that satisfy your query" and "These are all the rows that satisfy your query that you are allowed to see." But how much that matters and its impact on our application design we need to muse on for a while. There appears to be no feedback to tell you that your result set was constrained so we can foresee some interesting debug problems when a user reports that they only received X rows and your tests show Y but I guess we'll get used to it.

Column control is a little different in that you will receive a value--just not the real one. Because you can supply masking values for columns that have limited access rights, this is not much of a problem in handling character fields. We're all used to seeing things like XXXX-XXX-XXX-1057 for a credit card number in an email or report. Character columns can normally be masked with characters that could never otherwise occur in them, and if you wished, these could be standardized for all columns in the shop. As a result testing for a masked column is not that difficult a feature to add to an existing program that processes the table.

Dealing with numeric columns, however, could be a bit more of a problem. While most columns will have some limit (e.g., this column can never contain zero, another can never exceed 50,000, another can never be negative, etc.) they will rarely fit into a single pattern and so a standardized approach won't be possible. In some cases we might need to mask to a negative value, in others to all 9s, etc.

One thing is for sure, you cannot simply apply RCAC to an existing table and program combination and just "let it fly"--particularly if math is involved on the column in question. You need to think through the impact on your code and possibly make adjustments in the logic to accommodate the new situation.

We find ourselves thinking it would be nice if the database returned a status of some kind with each row/set to indicate that some data is "missing" due to RCAC considerations. After all, we've been increasingly using null capable columns to handle situations where a special value in a column just doesn't work. Having to go back to using special values makes it feel as if we have taken two steps forward and one step back, at least from a programmer's perspective.

While this new support is both important and useful, we do feel that the lack of any standardized way to readily detect "lost" data may be problematic. But maybe we just worry too much. What do you think?