The Sledgehammer – Version 2.0

March 11, 2012

The Basics of Software Testing: How Well Does Your Software Really Get Tested?

Filed under: Quality Assurance, Technology — Tags: , — Brian Lutz @ 1:14 am

(Note:  This post is part of what I plan to turn into an ongoing series of posts describing some of the basics of software testing.  Although there has been a lot written on the subject of testing over the years, I have found that a lot of it is, to put it nicely, not particularly user-friendly.  Boldly ignoring the adage I saw on a recruiting poster at Microsoft once saying you should work for a particular team because you never want your Mother to understand what you do for a living, what I intend to do here is explain, in a nutshell, how software gets tested, how the results of that testing get dealt with, and generally what the role of a software tester is in the development process.  I’m trying to keep the jargon to a minimum here, but I apologize in advance if this gets just a bit too technical.)

In our modern society, it can be very easy to take things for granted.  Take, for example, your household appliances.  Most of the appliances found in a typical modern household are items that just a hundred years ago were either years from being invented at all, or very expensive and difficult to attain for most people, with the tasks they perform requiring significant quantities of manual labor to perform.  In particular, it could be an all-day project just to do the laundry by hand in days of yore before washing machines were invented.  In our modern age, this once laborious task has been reduced down to throwing your clothes in a machine with some detergent and maybe a bit of bleach, setting a dial and walking away and throwing it in another machine when the buzzer goes off at the end of the cycle.  In fact, about the only time most people are going to give much thought to the process of doing laundry is when something goes wrong, such as when the machine breaks down for one reason or another, or when an imbalanced load creates a vibration in the machine that can shake the entire house.  In the same vein, the previous apartment I lived in before I moved to Downtown Bellevue provided me on a couple of occasions with firsthand experience with the joys of a hot water heater developing a leak at 8:30 on a Sunday evening and soaking the surrounding carpeting.  Can’t say I ever paid much attention to it before that one happened.

By the same token, much of the software that runs not only our computers but also an ever-increasing number of other electronic devices found in a modern household tends to fade into the background for most users 99% of the time, and only really gets noticed when something goes wrong or proves incapable of doing whatever that the user is trying to use it for.  When things start breaking, it’s easy for the average user to jump to conclusions and blame a lack of testing on the part of the developers for their present woes.  Although in some cases that assertion may not be entirely unjustified, it’s often a lot more complicated than that.  It’s been said that there is no such thing as a completely bug-free piece of software (with the possible exception of one of the “Hello World” programs that you tend to find on the first page of a beginning programming book,) and as programs continue to get more and more complex, there’s more and more things that can possibly go wrong.  Naturally, this also means that more and more things will, in fact, go wrong at some point.  And by the time all the development and testing is done and the software goes out the door, it’s inevitable that a few things will have slipped through the cracks, no matter how much testing gets done and how many testers you have hammering on it. 

What might be a bit less obvious to the average user is that there will also be a number of bugs in any product that were identified and filed by the testers, but were not fixed in the final product.  Some people might find it shocking that software would be shipped with the full knowledge that there are bugs in it, but it’s really just a natural part of the process of developing software, and 99% of the time, it’s nothing you would ever notice anyway.  I’ll most likely save the subject of how bugs are found and how they’re dealt with for another post, but when it all boils down, it mostly boils down to prioritizing things and making sure that the limited development resources on a project are being used in the most efficient manner possible, and at times a bug fix will prove to be more trouble than it’s worth.  For a novice tester this can be a difficult concept to understand at first, but over time they will learn to pick their battles and know what they might need to push back on and what they should just let go.  In short, not all bugs are created equal.  In a future post in this series, I plan to go into a bit more detail on how bugs are handled within the software development process, but in general, there’s some process of sorting and prioritization (commonly known as triage) that takes care of this aspect of things.

That said, when a product (or in most cases, a smaller portion of a much larger product) gets handed off from the developers to the QA team, how does it get tested?  Although a lot of the details are going to be dependent on the specific item in question, ultimately the approach is going to be similar no matter what you’re testing.  It is for this reason that a favorite technique of technical interviewers interviewing candidates for test positions is to specify some random item, and ask the candidate how they would test it.  Although most of the time this ends up being some sort of software component, there’s no reason this has to be limited to software, and on a number of occasions I’ve been asked how I would test completely random items ranging from vending machines to those misting systems they use to spray water  on the produce at your friendly neighborhood grocery store.  You’ll find in an interview for a technical position like this that the interviewers are looking less for “correct” answers to this problem than they are looking at the thought process that goes into it.  For even the most mundane and utilitarian of items, it should be possible (and expected) for an experienced tester to quickly devise a plan for testing many different aspects of that item, including functionality, performance, stress, real-world usage, globalization, and a number of other situations that particular item may be involved in.  Although the product in question is often something that has little to no relevance to the product that the tester is ultimately going to end up working with, the methods will be similar, so this still makes a good illustrative example.

When it comes to actually testing software, the approach here is going to be similar but at the same time somewhat different, and in my experience tends to be largely dependent on what is being tested, and where in the development process you happen to be joining.  For example, if you’re joining a project that has been in development for quite a while and which already has a test team in place, chances are that a lot of the groundwork of test plans, test methods and test cases will already be in place, and the duties of the tester will be largely in maintaining the existing base of test cases, running test passes, and (if applicable) working toward automation of the existing cases.  On the other hand, if a tester is joining in on a project that has had little to no testing done on it prior to his or her arrival, chances are that the tester is going to be doing a lot more of the foundational work of creating the test plans and the test cases on which the future testing will be based.  Not all the testing will necessarily be this structured though.  As with above, I intend to go into more details on how test plans and test cases are created in a future post.

Another common type of testing that takes place is known as ad hoc testing, which is basically the “bang on it with a hammer and see what breaks” type of testing that most ordinary people will think of first when the subject of testing software  comes to mind.  This can either be directed, where the tester is focusing on certain areas or certain features but still isn’t working from formal test cases, or undirected, where the tester is basically just messing around with things to see what breaks.  Even on a project with well established test plans, test cases and automation in place, there is still benefit to be gained from this less directed form of testing.  In my own experience, I have found that a good session of ad hoc is not only a surprisingly effective way to get started on something, as not only does it frequently catch a number of bugs and give the developers something to work on fixing while I’m in the process of establishing more formal tests, it’s also a good way to become familiar with the product and start planning for more formalized testing later on in the process. 

Although ad hoc tests can be a good way of  identifying a decent quantity of bugs quickly, they do come with some disadvantages.  For one thing, an experienced tester might be tempted to dig into deep dark corners of the product and try to find some really arcane bugs.  Scenarios like this are what’s known as edge cases, and represent the types of situations that are going to be encountered rarely (if ever) when real-world users start using the product.  Fixing these more arcane bugs will generally be a low priority for the developers, as they will often have far more important things to be working on.  Even so, the benefits of ad hoc tests tend to be worth the effort that goes into them, and on large software projects it’s not uncommon for what’s known as a “Bug Bash” to be scheduled somewhere in the development process.  A Bug Bash is basically a period of time set aside where developers, testers and even project managers all get together and just bash on the product (with varying degrees of direction, usually more generalized than specific) to try to dig up as many bugs as possible in a certain period of time.  Some places even go so far as to turn this Bug Bash into a competition with prizes for whoever finds the most or the biggest bugs.  This can be a useful tool for getting different sets of eyes on the product, and perhaps getting a bit out of the tunnel vision that testers can sometimes develop in the course of testing things.

Of course, a piece of software in development is a moving target for testers, and as a result, it is a necessary part of the development process to run the same sets of tests frequently to make sure that not only are things working the way they’re supposed to as they get developed, but also to make sure that further development on other areas of the product isn’t breaking existing functionality or introducing new bugs into existing components.  As a tester, running the same sets of tests over and over repeatedly tends to become incredibly tedious.  This is where automation comes into the picture.  Automation is another subject that will most likely get its own post later on, but in a nutshell, automation is the process of using a piece of software known as a test harness to automatically control the functions of the software being tested.  Since the vast majority of test cases that will be run in the process of testing a product are going to be specific sets of actions with  predictable expected results, this means that the test harness can be programmed to perform these actions automatically and verify that the resulting output of those actions matches what is expected, assigning a pass or fail result to each case based on this.  Having the software being able to essentially test itself can greatly reduce the drudgery involved in running the repeated test passes necessary in development, and is especially useful for the set of high-level tests that gets run for each build to make sure that it is acceptable for testing (known as Build Verification Tests, or BVTs.)  Although automation can greatly reduce the workload of the testers, there are limitations to what can be tested with automation.  For example, user interfaces tend to be a lot more difficult to automate compared to lower-level components like APIs and webservices, and it is virtually impossible to fully automate the testing of a piece of software because of the sheer complexity involved in most software projects these days. 

There are a number of off-the-shelf products on the market that can be used to create automated tests, but in my experience I’ve found that a lot of the places I’ve worked tend to reinvent the proverbial wheel when it comes to test automation, and it’s surprisingly common to see automation for a product being handled by a proprietary test harness made specifically for that product.  Because of this, the process of automating tests can require far more programming than testing.  Most test automation is created by a group of engineers known as Software Development Engineers in Test (commonly abbreviated to SDETs) who tend to exist in a bit of a fuzzy space somewhere between the development team and the test team (usually closer to test than development though,) and who spend a majority of their time programming automation tools and creating the automated tests that will be used.  In fact, it is not uncommon to find development teams where all the testers are SDETs, and comprehensive knowledge of software development and programming languages is an expected prerequisite to qualify for the work in addition to the usual test experience.  This isn’t to say that any project manager in their right mind is going to rely exclusively on automated testing to verify the quality of their product; in most cases this means that you have people who are wearing both the developer and the tester hats as needed.  There are quite a few other types of tests that take place in a software development cycle (such as performance testing, stress testing, internationalization testing, unit testing and others,) but the ones you see above tend to account for the vast majority of testing that happens on a product under development. 

As discussed above, people tend never to notice all the work that goes into testing a piece of software until something goes awry, at which point they invariably suspect that no matter how much actual testing took place, there obviously wasn’t enough of it.  And yet in spite of this perception, testing is a fully integrated part of the software development process, just as important to the success of a product as the developers who do the actual development work are.  As I continue with this series of posts and try to explain in more detail how software gets tested, I hope to give a little bit more visibility to the role of testers and just how important the work they do is in ensuring that the average user rarely has occasion to pay much attention to the software they are working with.  It’s a nerdy job, but someone’s gotta’ do it…

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: