Automated Quality Control, Part I

Quality control is critical to the success of a business, and in turn, to the success of its customers as well. This is doubly true in the case of software businesses and products, where problems and defects are rarely obvious. In this first post in a series about Snowtide’s approach to quality control, I touch on some of the specific challenges we face in ensuring product quality.

In PDFTextStream’s early days, before we were promoting it widely as a reliable, high-performance library suitable for enterprise applications, our quality control was weak. We had a few beta users, who stumbled across errors and PDF’s that caused PDFTextStream to fail. We had a few scripts that harvested PDF’s from various places (all of the PDF’s linked from a particular webpage, for example), which we could then test against PDFTextStream. For some time in the early days, our testing and quality control ‘procedures’ were . . . weak.

Most software, especially that which is built for some constrained, well-defined purpose, can be tested very readily. Start it up, click a few buttons, browse through a few pages, run some test data, maybe do some stress testing if you have time. Does it work? Yes? Great, deploy it. Otherwise, fix the problem, and repeat.

However, because of the nature of PDF documents, the fact that PDFTextStream didn’t fail on PDF #1 didn’t mean that it wouldn’t fail on PDF #2. Further, just because PDFTextStream didn’t fail on PDF #10,000 didn’t mean it wouldn’t fail on PDF #10,001. Since there is no notion of validity when it comes to PDF documents, there is no way to say that PDFTextStream is Fully Tested™. This is completely different than the circumstances one might find when testing an XML parser, or a web server, or a web application, or an accounting utility, where the inputs and outputs are well-defined and specified.

Our circumstances are, however, very similar to those I would imagine exist when testing a new car, or a light bulb, or an iPod. There, you can only hope to minimize the likelihood of failure, since the nature of the product is such that it will fail eventually. Similarly, PDFTextStream (or any other PDF library or viewer) will never be able to claim that it will never emit an error, because the PDF document format inherently allows for a wide degree of variability.

Once you move past the notion of proving the lack of defects in a piece of software to accepting that the attainable goal is to minimize the likelihood of defects as much as possible, there is a very straightforward approach to quality control: test, test, test. Not to eliminate the possibility of faults, but to minimize their occurrence to a reliable, quantifiable percentage of the total tests performed.

So, it is with this focus that we embarked on building an automated quality control environment. Our plan was simple: build a system that will test PDFTextStream against astronomical numbers of PDF documents — orders of magnitude more than any customer would ever dream of processing — then see where PDFTextStream fails, and fix the problems.

Over the next few weeks, I’ll be posting the highlights from our experiences rolling out this automated QC environment, as well as some notable instances where failures discovered by this process have prompted significant improvements in PDFTextStream.

Update: Part II is here.