Happily Waiting for Reports, Not!

In a former life I was a user in between stints as a programmer. Although I would say the experience taught me humility and respect for those programmers who I requested things for, I would be lying. Every time I would ask the programmer why this report was taking a lot of time, the response would invariably be: "Do you know how much data you are requiring me to sort through to get you what you wanted?" Of course, I played ignorant and said: "It just a few million records, it should not take hours."  The next reply from the programmer was: "If you think this is so easy, why don't you write it yourself!"

A year ago I was helping out on a simple report that should run in a few minutes. It took 1 hour and 40 minutes to run an accounts receivable aging against 2 million plus records. First I looked at the select and found it was taking about three to five minutes to get all 2 million records. I changed the select to use alternative keys and reduced the criteria to select against.  Those coding changes reduced the select time to under 3 seconds.

The code that processed the data was building JSON arrays that are passed on to the open source report writer from Eclipse.org called Business Intelligence Reporting Tool (BIRT). Building those arrays took about 1 hour and 30 minutes using a standard UniBasic code structure of appending to a dynamic array. I could not change the code to a dimensioned array because the size of the report varies from 1 pages to over 4000 pages. I tested it against my better judgement with a dimensioned array that would cover a 5000-page report and it did run in about an hour. 

There is a relatively new structure in UniBasic called UDO which stands for U2 Data Objects. I created the JSON arrays directly using the UDO function calls. The time to process the data was down to 50 seconds. Wow, that was more improvement that I had anticipated. Our BIRT report in production finishes in a little over one minute.

Maybe your programmer is right that it takes hours in today's world to process large amounts of data. Or maybe your programmer is using outdated coding technique that can be improved by being a lot more knowledgeable.

The Feedback Loop

When we hired a new employee a few weeks back, we always talk how U2logic is customers service oriented company. U2logic develops many types of applications and tools and we believe in the constant feedback loop between ourselves and our customers.

Years ago we tried to writing our own ticket tracking system. We spent a lot of time and did not get it right. We abandoned that project pretty quickly and went back to having our customers email or call us with issues. We would then figure out if the problem was code related, data problem, or a training issues. If it was one of the first two we would send off an email to our customer explaining the fix. If it was a training issue, we would try to email the steps to fix the problem or worse case talk them through it on the phone.

We knew our email/phone system was very marginal at best so we began to look for alternatives. We found JIRA Software the Issue & Project Tracking for Software Teams from Atlassian. We slowly started implement the features and feedback we wanted and got our staff familiar with the product. We bit the bullet and added our customers to their respective projects.

Wow, this was a surprise in how much our customers want to be part of the process. We get many request per day for problems, questions, or new features. Sometimes the response looks overwhelming to our new employee's but we love the feedback good or bad or indifferent. Our new employees think we bend over backwards for our customers, we hope they feel the same way as we are creating fantastic software.

Pebble Theory of Coding

There is a story of a man on a beach trying to figure out how to move just these large pebbles scattered all over the beach and leave the sand. This man can get a big front-end loader to remove all of the rocks.  Or this man can hire 100 helpers with wheel barrows and remove the rocks one a time using manual labor. Or, lastly, this man can remove several rocks every time when he visits the beach.

Most programming problems are based on the pebble theory of coding.

We can buy a very expensive tool to look at code to find our problems as well as help us develop code. This tool's purchase will have to be justified as to why it is needed with management which will be an arduous process. This tool will do the job but might report back false positives during code review.  Additionally, this tool may have a huge learning curve that will cause very few programmers to take the time to learn and use it. The question remains: is the beach, in this case the code, still usable.

This is what large or small shops do.  Given the size of the programming project, they may hire 10 or 100 programmers to do this task.  They give them cheap or open source tools to do the job because they have spent the money on labor. The project gets done in a few months or years depending on management changes and direction. The question remains: is the beach, in this case the code, worth the labor cost to get it done.

A single programmer over the history of programming has always created better and more innovative code than groups. Part of the reason is that a single programmer does not have group focus but a single focus on the task at hand and is able to get from point A to Z via D and M rather than going through the whole alphabet. Even this singularity may produce bad results or incomplete code but on balance should produce a better software product because their is only one person to blame.  The question remains: is the beach, in the case the code, worth it since the code might be much smaller and consistent.

Moving from Telnet to Web the Truth Revealed

Moving from a telnet based interface to a GUI like a web browser is always a tough decision for most U2 shops. However, most vendors sometimes "Lie by Omission". Let’s take apart the decision and talk about what the other vendors say or don’t say about this processes.

Web browser interfaces is slower than telnet. When your people can open several browser tabs to see information that would take them multiple enter keys to find in telnet that argument can be rendered moot.  The other issue is data entry speed is better in telnet. If you have written the user interface (UI) in JavaScript, there is no latency speed issue to go to the server because most of the data entry correction and verification is taking place on your web browser. For the most part, your vendor can, with a lot of work, negate this issue.

Converting from telnet to web is quick. Most if not all of your logic in your programs is based on receiving information from each telnet prompt. No web interface works like that.  At U2logic, we try to send either an entire multi-valued line of information at once or the full screen to do the database integrity testing at the database level. The logic of your programs for a different UI’s means removing a lot of code and restructuring the flow to be more batch mode instead of interactive mode which will take quite a lot of time to get it right.

Web programs will save you money in the long run.  This one is very true. Once you have made the jump to a web browser your business can grow without the need to hire more people because of productivity gains.  Additionally, when turnover happens which is expensive, you training time is significantly reduced for those new employees.

U2 Reporting and Saving Code Lines

Over the many years we have been writing reports for clients in UniBasic, we have never seen the requirements that all reports need to be in PDF or XLSX format. Historically, all of our reports have been written in UniBasic code and outputted to the web. Our users then could change the output from HTML to PDF with a free tool. We would even add UniBasic code so our reports could be outputted to CSV format that could be converted to XLSX. That worked for about 10 years but every time we sit down with a client for a new report they only want it in either PDF or XLSX.

At U2logic we build tools using Eclipse IDE.  We had looked at Business Intelligence Reporting Tool (BIRT) many years ago to give us access to a tool that used by over 2.5 million programmers with over 47,000 active members in their community. Within BIRT is an open data access (ODA) way we could have BIRT talk to our database with normalizing the data. We build an ODA driver that takes either a query statement or can call a UniBasic subroutine to get the data necessary for BIRT to work.

We have converted around 75 reports to date using BIRT and our XLr8Reporter module in Eclipse IDE. What we found surprising was that we saved code while giving our users the report formats the wanted. The total code saved so far is around 20,000 lines but our reports are fully functional and look great.  The other benefit is that when our clients want to move data around the report or add break totals, it takes us minutes instead of hours.  We have never had a problem with a report not footing correctly because BIRT handles all of this.

Very few times in our coding experience have we had a win-win when we add new technology.

Drive-By Programmer

Are you a drive-by programmer?

What this means to your colleagues is that you just offer a suggestion to fix their code and move on. In order to get new programmers to use UniBasic you have to spend the time teaching, critiquing, suggesting and even writing a snippet of code to help them get over the hump or the might just get another job outside of Universe, Unidata, D3, or QM (U2).

Teaching those junior programmers and even senior programmers who think they know it all requires understanding what they are doing and where they want their code to do. Additionally, you must show them what tools are available to make their job easier and more productive.

Junior programmers probably have heard and used Visual Studio and want a tool that allows them to learn and grow on the job and we have that with Eclipse IDE for U2. Once they see their compile errors as the type, or hitting F3 to see what these includes are they will be much more of a productive U2 programmer. Maybe after we have taught them that every variable in U2 is global in scope and they have not passed out or told you how crazy that is, you can show them in Eclipse IDE Outline at what line number these variables are used at.

Senior programmer can be shown that using line editor or full-screen editor where they have to either a telnet session running in background or have to FTP the source code back the database, that there is a better way. With Eclipse IDE you can directly edit your source code on the database without all of that hassle and be able to produce code much faster and cleaner.

Whether you are a junior or senior programmer saving off all programs at BP.OLD or with a prefix like MYPROGRAM.0415216 is such a poor way of handling changes that we look like we started coding the in the 1980's. Oh, we did. Version control using GIT, TFS, SVN, CVS, or whatever is the way to go. Even within Eclipse IDE there is history of all of your changes throughout the day so you can even go back to the 9 am version at 4:30 pm.

A trial license of XLr8Editor for U2logic or U2 is available for free.