Bret Pettichord
bret@pettichord.com
Copyright 2001, Bret Pettichord
Presented at STAR West, 31 October 2001, San Jose California
How do you know if you are ready to get started? And if so, how should you get started? Who should do the automation? What approaches should you take?
At one of these conferences a couple years ago, a testing consulting company was handing out some brochures describing an engagement. They were selling automation services for tool X. Their brochure described how they came in and initially solved some technical problems getting tool X to work with their software. These problems were problems that had been impeding their automation, so this was great. And then they went on to automate tests. But then they started to realize that they didn’t really know what tests to automate and they had to take a step back and help the organization figure out what tests they need to run. They’d had an ad hoc process that needed to be systematized before it could be automated.
I thought it was really a great story. It was all told in a one-page flyer that purported to tout their technical skill with tool X. But just barely below the lines, they were actually admitting that the tool problems, which were real enough in themselves, weren’t really what they client needed help with. But I suspect that their test manager had an easier time convincing his manager that they need test automation consulting than admitting that he needed help getting his work organized.
If people are dissatisfied with testing, it is easy to say that the solution must be automation. The problem may be that the developers aren’t doing enough to check their own code. The problem may be that the testing is disorganized. The problem may be that the testers aren’t getting the information they need from development. But addressing any of these would require that the organization admit that some people need to change the way they work. This kind of change is difficult. It is easier to just say that what you need is automation.
I got a phone call asking for help a couple months ago. It was from a development manager at a small software company. They had been testing their software informally, with developers checking each other work, but their customers had been complaining about poor quality and he’d been charged with formalizing their testing. He’d bought a copy of a testing tool and was calling me to get some advice about how to use it. Did he have some dedicated testers? No. Who was going to use the tool? He was. This was someone who was in serious denial about his situation. A test tool wasn’t what he needed. But he wasn’t ready yet to spend what it would take if he was going to put together a serious testing capacity for his company.
In my consulting work, I have focused for many years on test automation. But I have had to learn more and more about testing in general. One of the reasons is because I was seeing over and over again that problems were being labeled test automation problems when, technically, they weren’t automation problems at all. It is just that test automation is more available, non-blaming labels available when you have problems in testing. A term for this is say that this is the “presenting problem”. You’re not really ready to automate until you can tell whether it is a presenting problem or not.
I’d like to talk about some work I’d done with a client. They’d brought me in to help with the test automation as well as to help with the testing strategy in general. This was a project that had a lot of problems. It was a big development project that was much larger in scope than the organization was used to dealing with and consequently there were a lot of problems. They were used to a lot of informal processes that just weren’t scaling up. Part of this was that they need to formalize some of the hand offs between development and testing. They also had issues with requirements and what not that I was helping them with. I was also working on the automation strategy. Most of it was still conceptual, although we did have some working code.
Later I got some feedback from the client that the staff had been unhappy that I hadn’t done more to help them with automation. It was interesting feedback and it took me a while to figure out what was going on. Whenever I say upstream problems that were going to prevent automation from being effective, I would work on those problems. Sometimes they were development problems. Sometimes they were testing problems. Sometimes they were really problems with the scope definition of the project as a whole. Partly because of my help, the testing group did a good job of testing software when it was delivered to them. They found important problems and turnaround was good. So I had judged that things were working pretty well. But then I realized that the testers were looking for something different.
They weren’t looking for automation to improve their effectiveness. Their effectiveness was fine. They were looking to it as a way of building status, both to themselves and to the programmers they were working with. And I wasn’t helping them with this at all.
Indeed, when I look back at projects I’ve worked on, I realize that some of the best test automation people I’ve worked with have not had their egos tied up in their work. Many had already proven themselves as developers and they were more concerned with providing value than demonstrating prowess.
Test automation is often assigned to junior programmers. I think the idea is that it gives them something to do that will be useful if it works and is low risk if it fails. The problem is that invariably we have people who have learned a lot about programming in college and are looking for a chance to prove what they can do. They also probably know very little about testing, but since they are junior programmers they don’t know what they don’t know. So they end up creating automation that is overly complex, yet fails to meet testing objectives.
There are two different philosophies of test automation. In one, you try to reduce the intrusiveness of the test software on the product software. In the other, you try to find opportunities to place hooks into the product to facilitate testing. At first glance, this looks like a technical issue. Is adding test software going to “pollute” the environment in a way that might invalidate the test results? I’ve collected a lot of detailed information about how test tools instrument software. What I have found is that over the years, the degree of impact hasn’t really changed, but test tools have become better and better and making these intrusions easier to apply, often happening now during installation without announcement. Whereas they used to require that your developers compile code into their binaries, often inserting calls directly into the main event loop.
I now believe that many of the concerns about intrusiveness don’t really stem from the valid concern that changing the test environment could theoretically impact the validity of test results. Rather, the argument is used by developers who don’t want testers to intrude in on their work. They want to limit their interactions. They don’t want a lot of questions and they certainly don’t want testers messing with their code. This is where the heat behind the intrusiveness issue comes from.
Needless to say, I have found example after example of products that were easier to automate and were better tested because well-designed intrusions were placed into the product to make it easier to test.
I believe that one of the reasons that GUI test automation is so popular is because it appears not to require changes to the product software. But this isn’t really true. Successful GUI test automation often requires or is significantly improved by making changes to the product code.
Like I said, the intrusiveness issue really comes down to a matter of how testers and developers relate, rather than a policy regarding code. Testers who are sealed off and who are expected to treat the entire development organization as a black box that emits product software and accepts bug reports are at a big disadvantage. Their testing will suffer. They may even conclude that the solution is automation, but effective automation will require even closer collaboration between testers and developers.
In my consulting, I really try to find ways to get testers and developers to work together on their automation strategies. All kinds of shortcuts can come of this. Developers often know of easy ways to meet testers goals. But they need to understand them, and they need to feel like the testing problems are something it is worth thinking about. At the same time testers need to know how make effective use of developer involvement. I’ve seen too many testers who just whine about their problems to the point where they can’t even notice when they have someone who would help them if only they would stop their whining.
I am a big believer in iterative test automation. Many of you have probably heard about Extreme Programming, which is an iterative programming methodology that has been getting a lot of press, and there are some talks at this conference that will be talking more about this.
One of the problems that I’ve seen with the attention that Extreme Programming is that they explicitly say the methodology only works with small teams, say under a dozen people, and that they require onsite customer involvement. The projects I work with are usually bigger and they usually don’t have onsite customer support. So Extreme Programming doesn’t apply.
However, test automation is a perfect candidate for Extreme Programming. I’ve never had an automation team with more than 12 people and they’ve always been located with the testers and developers who are the customers of the test automation.
I see several benefits to an iterative approach.
First, it’s hard to know in advance which tests are going to be good tests and which aren’t. In my view, good tests are more likely to find problems. Sometimes you might have contractual requirements to automate certain tests; when this happens, you may not need an iterative approach. But more often, it is only with experience of the software that you know which ones are worth automating.
Second, by definition test automation must adapt to the product software. So it is hard to know in advance what approaches will work and which won’t. An iterative approach is an effective way of figuring this out.
Third, an iterative approach helps you understand what the important requirements for automation are and whether they are being met. There are a lot of unstated requirements. Some people want automation to save cycle time. Others want to reduce testing cost. Still others want objective measurements of progress. An iterative approach will help you draw out these requirements and understand better what your customers want.
I find that many automated tests fail because they don’t meet the required quality criteria. The functional requirements for automation are usually clear: this is the testing that the automation accomplishes.
But other quality criteria can be missed. These are things like maintainability, robustness, integrity and reusability. I’ve seen cases where automated test suites become unusable because of problems meeting these kinds of requirements. Ultimately your test suites must not only run, but they have to actually execute the tests that they claim to run.
James Bach tells a story about a test suite that he’d inherited in one job. Everyone thought it was doing a great job of regression testing the software until they tried to run it once on a machine that did not even have the product software installed. Only nineteen of some 200 tests failed. Clearly there were some requirements that were not being met by this test software.
An iterative process will help you understand various quality attributes that your test automation must have.
Here’s two definitions of automation.
(1) Automatically executing tests that would otherwise be manual.
(2) Finding creative ways to mix programming and testing.
Let me just come out and say it. If you have a fixed idea that automation is finding ways to automatically execute tests that otherwise would be run manually, then you are not ready for automation. Now it is certainly true that some people have gotten this to work. But it is a hard approach and often quite expensive. And it overlooks a lot of other approaches.
Let me lay out several dimensions for categorizing test automation.
(1) Interfaces. Tests can be automated to test software at the unit, API or system level (which is usually a GUI).
(2) Risks. Classically tests are automated to manage the risks of regression bugs or scaling problems. But automated methods can also be used to address risks related to concurrency problems, error handling, feature interactions, and resource waste (e.g. memory leaks)
(3) Test Creation. Some approaches to automation are meant to help non-programming testers create executable tests. Others are suited for tester/programmers who can write code. Still others use automation to automatically create tests based on testing models or code inspection.
(4) Evaluation. How do the tests know if the software is correct? How do they access the results and what are they compared to? Often the actual results are read off the screen and compared to previous values. But other methods exist to find problems. Logging mechanisms can be used to expose behavior. Asserts can be added to notify you when the software is in an invalid state. Flight recorders can capture data about the state of the system when it fails. And certain automated techniques allow for independent computation of correct results. All of these techniques can increase the effectiveness of both exploratory and random testing techniques.
(5) Coverage. Instrumentation in either the product or automation code can record what’s been tested to give you a better understanding of the coverage of your testing.
If you see these dimensions as defining a five-dimensional space, then you realize that one point of this space describes a paradigmatic case: GUI system test automation focused on regression testing for use by non-programming testers using previous versions as an oracle and without much attention at coverage.
There is nothing wrong with this paradigm. I have done it myself, and there is a lot of good advice out there about how to do it effectively. But some people see this as the only type of automation. There are tools to support it, and there is a wide need for it. But if this is your preconceived idea of what automation should look like for you, then you are apt to miss opportunities and inclined to commit to a style of automation that is often more difficult than people expect.
Lastly, test automation requires leadership from someone who understands both programming and testing. This is the only way you are going to be able to effectively navigate through all these possibilities.
Now I’ve given you a list of seven preconditions for automation. Now I don’t think I’ve ever seen a project the fully met all these preconditions, and I’ve seen plenty of successes. But the more obstacles you have the harder automation will be. If you have more than a couple, I would suggest that you go slow with automation and avoid big commitments.
About the Author
Bret Pettichord is a consultant who helps teams improve their testing. He is a co-author of Lessons Learned in Software Testing (testinglessons.com) and editor of the Software Testing Hotlist (testinghotlist.com).