« Happy Birthday, Mac | Main | Offer Alternatives, Don't Whine »

February 05, 2014



Thanks for another interesting article!

Our shop has no testing plan at all. The developers decide which tests they perform by themselves before their programs go to production and which tests will be performed by the user.

We code for our own use only. If other parties are involved (data transfer to/from customers or third party applications) we usually just figure out some tests and it works quite okay.

In my experience, our software is not always worse than the pieces of our partners (which are often enough commercial software companies).

However, I'd like to learn more about testing tools and a more planned approach to testing.

Paul Rogers

Jon and Susan:
Excellent topic and one area that I think has been inadequately covered by both organizations and the community.
I spent many years performing product testing for an ISV and the answers to your questions are as follows:
1. We always had a formal process in place, from unit testing to QA testing to Regression Testing.
2. There was no formal unit testing. The developer decided, based on the complexity, the extent to which he wanted to test to ensure he had all areas of code covered.
3. For a period of time we used an automated tool but at the time the tool was immature and buggy itself so I would spend more time testing the testing tool and reporting bugs than my job.
4. Code coverage would be attempted in unit testing based on complexity of change and all code was put thru Regression testing in an attempt to cover as much code as possible.
5. There was a formal Regression testing phase to all code changes included in a release cycle.
6. See Note below
7. ISV

The developers were divided into 3 categories:
1. unit test quickly and then send over to QA for more integrated testing and would always listen to my suggestions on how I thought it should work and might even see my side of the story on occasion
2. Unit test thoroughly and send over to QA for more integrated testing but always had the answer as to why it worked the way it did and only occasionally would see things my way.
3. Unit test thoroughly, and then unit test thoroughly again and I rarely found any issues to report on, and sometimes would see things my way.
#1 probably put out more work than anyone but occasionally had to go back in and rework something. His speed was phenomenal because if you brought an issue to his attention he had it fixed by the time you got back to your office.
#2 was able to see the product from a very high level and understood the ramifications of all changes(even ones he wasn't working on).
#3 took great pride in his work and never wanted anything to get past his desk without a thorough test.
... #3 was the best programmer to have from my perspective because he made my job so much easier when he tested so thoroughly.

Thanks for bringing the subject up and I would love to see some follow up discussion based on the feedback you received.

Von Enselman

I have been involved with QA/Testing specifically on the i since 2005. Initially I worked for a tool vendor so was responsible for training and implementing and IBM i specific testing tool for over 35 companies. What I learned is that there are as many ways to test in our world as there are to develop.

Unfortunately in more resent years I have noticed many shops getting more committed to testing protocols and QA groups/departments BUT not involving the developers sufficiently.

I would like to see more education on testing and QA concepts being marketed to IBM i developers. I would also like to see QA professionals in IBM i shops availing themselves of the tremendous education on the systems, languages, and concepts they test via our community. Personally I have found involvement in my local user group and COMMON to be a substantial asset in my testing skill set.

Thanks for talking about this important issue,
Yvonne Enselman, CTAL-TA
(Certified Tester Advanced Level- Test Analyst

Vern Hamberg

Just a few things that come to mind -

1. Original Software has a test suite that is very much IBM i-focused. Has been around for years.
2. I used to work for a place that used Aldon - Aldon had a component that would instrument your code in order to process coverage - basically it put something inside every conditional block to see if it got a hit.
3. Lately I've been a fan of the IBM Rational tools. At my previous employer, we started to use just Team Concert to gather requirements. Rational do have a specific requirements tool - a couple, actually. One is Requirements Composer, DOORS is another. And I believe they used Team Concert for storing test plans. In TC, all these artifacts can be connected and observed and all that.

There is a free offering of Team Concert - 10 licenses. So if you have a smaller team, this could be a useful tool.

I've no theories of testing in mind today - just tools!


DZONE has an article: Programmers Without TDD Will be Unemployable by 2022

I seldom see RPG that is modular enough to be unit tested. When coding for the 5250 display RPG is removed and the RPG is called from a web ui, the RPG becomes more modular. Generally, RPG is tested with the debugger and I was recently advised to test-debug a program in production.

Henry James

testing and QA is a huge subject - and way too important to leave to chance, so its good to see some structured attention being given to it.
and timely - I've been banging a drum about the QA we do here. web / dotnet gets all the attention, and IBM i a cursory glance, whereas all the systems in the world depend on a databse, and accurate transactions going across it. We have IBMi related tools - but they are too 'special' for the QA people to use, because they haven't an understanding of databases in general and DB2 for i in particular. Desingers and developers don't produce specifications detailing the actual, on the disk transactions that must, or must not, happen. RPGunit has its place, and especially so in the ILE world.
But, as you seem to have found, the starting point is to bring the issue into the light.
I'm looking forward to reading the conclusions.


We have no formal test doctrine here. Individuals determine if/what testing is sufficient. End users may or may not be involved, depending on the magnitude of the change.

As a rule, we do no formal unit testing. I myself use RPGUnit in conjunction with RDi to put new subprocedures under test but no other developers here see the ROI.

Testing interactive applications is very ad hoc. No automation aside from sometimes using Auto Hot Key.

No code coverage tools. No goals, for that matter.

No formal regression test plans. I have a home grown utility I use to compare the differences between two files; a copy before and a copy after the update process. It's only barely automatic - I still have to eyeball the actual deltas to see if they match what I want.

We develop for in-house use only. Everything is home grown.

Carl Novit

We are a small shop and our testing is by no means perfect. We use Aldon as well and have it structured so that we have Development, QA, and Production.
In QA, we created a test library and repopulate the library weekly with current data. From a business perspective this helps us look for meaningful results.

David Shirey

Certainly testing is important but I don't think you can compare the i and many web based languages testing wise. Just being a compiled language picks up tons of errors that would otherwise need to be 'tested' for. Likewise the strong typing of the i data elements. It is just hard to compare the two environments. Our testing is more oriented around the business objective and whether we are meeting that, not whether there are bugs in the code (although that is a factor too).

Dan Devoe

Interesting topic...

Jon & Susan, you are aware of my situation... I'm the only programmer in our shop, and as such, have no formal "rules" for testing, other than to make sure it doesn't royally mess up our other programs, data, and internal business procedures.

The last one is a touchy subject, because people often don't think of how a systemic change affects the rest of the company. So, as another poster stated, I essentially need to know everyone's job. But I digress...

Usually, I'll test changes in a different library, against data that is refreshed weekly.

But there are times when we simply can't do that... I just ran into an example of it this past week.

Added a pf trigger (*INSERT *AFTER) against our order header file (table). Trigger program was tested, and first issue found involved activation groups... our base software is still RPG/400, and all stuff I'm doing is RPG/LE. Rather than trying to fix the activation group issue, I made the trigger program use ACTGRP(*NEW). I know this isn't advised, but it fixed the issues we were having, and the amount of orders being written won't degrade performance noticeably.

Ran it thru a bunch of tests as described above, and got the ok to put it live.

So it was put ive. Everything worked great, until one particular program (our off-line order entry program, which deals with EDI, B2C integration, etc) started doing some "funky" things. This was a case where this particular program couldn't easily be tested in our "training" environment, so I dealt with it (issue resolved).

And as you also know, I still do a lot of work with VRPG, so that introduces a whole new set of challenges.

I'd love to chat with you move about this during the upcoming NEUGC

Ron Newman

Testing Schmesting. When it comes to good testing procedures you have to start with clear and well developed requirements. Once you have that you can build test scripts for all phases of testing. Unit testing should be preformed by the developer saving test scripts and results. QA testing should create test scripts using the requirements as a base to make sure the developer met the requirements. No matter how small the change to the program is full regression test must be preformed.
The shop I work in fallows the above format. It adds time to the development cycle however in 99% of the cases the product works in production as required. Which is our goal as an IT department

Andrés Gómez Casanova

I wanted to share with you an open source project I developed. It is a unit testing framework for DB2 LUW, that probably with some modifications it could work on DB2 for i.

The project is completely developed in SQL-PL, and it aims to tests the native data types and return a report. It is a xUnit framework, and could be considered as a jUnit porting to SQL-PL

If you want to take a look, the project is hosted at

Comments are welcome in the issue section.

The comments to this entry are closed.