Success with Test Automation

Bret Pettichord

BMC Software

Personal Background

• GUI Test Automation Projects
  • Interleaf I 1992
  • Interleaf II 1993
  • Snap/Segue early 1994
  • BMC MetaSuite 1995 -
• 8 years QA experience
  • Commercial applications
  • Open systems

Overview

• Test Automation Perspectives
• Getting Started
• Two Key Design Concepts
• Automating BMC-Meta Testing
• Living with Test Automation

Planning test automation

1. Select a test tool

2. Record test scripts

3. Run scripts and test product

Don't be fooled. It's easy to create throwaway test scripts.

I.Q. Test

Which can you do in your spare time?

A. Automate Software Testing

B. Automate Employee Payroll

C. Automate Manufacturing Process

D. Automate Airline Reservation System

E. Coordinate Company Picnic

Don't try to automate testing in your spare time. Schedule and dedicate people and time.

Test Automation is
Software Development

• "Test tools" are really test development environments.
• It is a 'development' activity, not a 'test' activity.
• It needs people who understand how to design, develop and maintain software.
• Source code control, design review, code review . . . Do it right!

Thinking Smart

1. Choose good people, get good training.

2. Early success, proof of concept.

3. Automate the right stuff.

4. Plan for maintenance.

Don't be fooled. This requires planning and feedback.

Getting Started

• Get good people.
• Use software engineering principles.
• Plan to reduce maintenance.
• Carefully pick what to automate.
• Get first results quickly.
• Automate your acceptance tests.

Get good people

• Automators need to
  • understand testing requirements
  • know how to develop software
  • maintain test system as product develops
• Contractors
  • How are you going to maintain the tests?
• Developers
  • Do they understand testing requirements?
• Testers
  • Can they develop software?
• Don't skimp on training
• Don't accept rejects from other projects

Use Software Engineering Principles

Test automation is not a silver bullet

• Must be maintained
• Must be tested
• Use source code control
• Review designs before coding
• Code reviews: good idea.
• Use feedback, spiral development
• Usability is important!

Plan to reduce maintenance.

• 70% of software development costs occur in maintenance.
• Anticipate User Interface changes.
• Organize code to avoid duplication.
• Inform developers of consequences of UI changes.

Carefully pick what
to automate.

• Automate based on manual tests.
• Things you will run many times.
• Assume automation will take 10 times as much time.
• Avoid automating tests you wouldn't run manually.
• Ignore performance
• Don't be a perfectionist

Get first results quickly.

• Identify testability issues early.
• Accept partial automation.
• Validate your testing strategy.
• Generalize your test scripts later.
• First just get one test case to run.

Automating Acceptance Tests

• These are the tests run most often.
• Include a couple tests for each area of the application.
• It's boring to run the same tests over and over. Bored testers miss bugs.

Two Key Design Concepts

Test Driver Architecture

Error Recovery System

Test Driver Architecture

• Assume Automator and Tester are different people.
• The Automator creates the test driver.
• The Tester creates the test cases.
• The test case file specifies the details of what to test.
• The test driver makes it happen. It knows how.

Test Case Files

• Independent of implementation details.
• Only need to change when product requirements change.
• Can be written before application is ready for testing.
• Different format for different Test Drivers.
• Requires no knowledge of test tool.
• Presumes understanding of product requirements.
• Can be created and understood by domain expert.
• Format is well-documented

Test Drivers

• Interpret test case files and drive the application.
• Written in test tool language.
• Need to be maintained when UI mechanics change.
• Can include verification.
• Requires programming ability, knowledge of test tool.
• Can be created by someone who is not a domain expert.
• Report errors in testcase data.

Advantages of Test Driver Architecture

• Easier to maintain
• Allow multiple contributers
• Use different people's strengths
• Clear ownership
• Division of labor

Problem: Cascading Failures

1. One test fails.

2. The product is left in an odd state.

3. Subsequent tests automatically fail.

Tests must be run and rerun

Error Recovery System

• Prevents cascading failures
• Allows unattended testing
• Places product in "base state" before and after each test case.

Building an Error Recovery System

• You need this even to debug your test drivers and test cases.
• Don't try to engineer it to recover from all "possible" errors.
• Instead, make it work with the errors you are actually encountering.
• QA Partner comes with a configurable recovery system.

Testing MetaManage

Using Automation

BMC Products: MetaSuite

• Family of Database Admin Tools
• Client/Server
• Client: Win 3.1, Win95, Win NT, OS/2/Win
• Server: Solaris, AIX, HP-UX, OS/2
• Databases: Oracle, Sybase, DB2, Informix, ...
• TCP/IP connectivity

Test System Components

Tester's Perspective

• Testcase files
• Test data
• Test suites
• Baseline results
• Test drivers
  • provided

Test System Components

Automator's Perspective

• Test driver
• Output verification
• Error recovery system
• Window declarations
• Custom class support
• Common functions and methods

Example Testcase

-- otbed001|This testcase should use the ALTER TABLE command to edit

-- otbed001| the tables storage values

-- otbed001|OS:

-- otbed001|TF: otbed001.001

-- otbed001|RI: TBED0095

-- otbed001|RI: TBED0077

-- otbed001|RI: TBED0086

|A |otbed001 |E |TB | |SA1 |TB01 | |E |000 |pct_free |10 |99

| |otbed001 |E |TB | |SA1 |TB01 | |E |000 |pct_used |40 |1

| |otbed001 |E |TB | |SA1 |TB01 | |E |000 |next_extent | |1

| |otbed001 |E |TB | |SA1 |TB01 | |E |000 |max_extents | |100

-- | |otbed001 |E |TB | |SA1 |TB01 | |E |000 |new_extent | |5K

Test Data

• Application Specific
• Our test data are database objects

Test Suites

11/15/95/ antbed.s - file contains only table edit testcases

antest otbed001 otbed001.001

antest otbed002 otbed001.001

antest otbed003 otbed001.001

antest otbed004 otbed001.001

antest otbed005 otbed001.001

antest otbed006 otbed001.001

antest otbed007 otbed001.001

antest otbed008 otbed001.001

antest otbed009 otbed001.001

antest otbed010 otbed001.001

antest otbed011 otbed001.001

antest otbed012 otbed001.001

antest otbed013 otbed001.001

antest otbed014 otbed001.001

antest otbed015 otbed001.001

antest otbed016 otbed001.001

antest otbed017 otbed001.001

antest otbed018 otbed001.001

antest otbed019 otbed001.001

antest otbed020 otbed001.001

antest otbed021 otbed001.001

antest otbed022 otbed001.001

Example Test Driver

• "Analysis" test driver
• Tests core function of MetaManage product
• Generates workscripts based on specifications
• 2300 lines of code (4test)
• Several months of development time
• 1055 test cases

Output Verification

• Not needed for all test drivers
• Ours is a shell script
• Compares results against a baseline
• Ignores timestamps and other data that we expect to change

Recovery System

• Detects and reports errors.
  • scripting errors
  • application error dialogs
  • application does something unexpected
  • application crash
• Resets the application for successive test cases.
• Allows unattended testing.

Window Declarations

• Requires knowledge of GUIs.
• Maintenance
  • Some ui changes require no maintainance.
  • Others require simple changes.
  • Related dialogs use common class ancestor to simplify maintance.

Custom Classes

• Allow test scripts to read and drive custom controls.
  • Grid (Spreadsheet)
  • Outliner (Tree list)
• Techniques
  • Key Events & Clipboard
  • Call Windows DLL (Send Messages)
  • Directly call API (Extension kit)

Layered test components

• Easier to maintain
• Allow multiple contributers
• Use different people's strengths
• Faciliates re-use

Automation Accomplishments

In a week in September, we ran

310 unique "analysis" testcases a total of

1030 times. Of these,

127 were run once;

50 were run 2, 3 or 4 times;

80 were run 5 times;

53 were run 6 to 10 times.

Living with Test Automation

Committing people to test automation.

Avoid false positives.

Accept "Good Enough" test automation.

Test Failure Triage.

Commiting people to test automation

• Requires different skills from testers.
• Multiply tester's productivity.
• Can focus on where the big wins are in test automation.

Avoid false positives

• Test Automation will have bugs.
• Here is the most serious problem you could have in test automation:
  • Report that the test passes when it really doesn't.
• Take all precautions to avoid this.
• Most common causes:
  • Exception handling
  • Switch statement without default

Accept "Good Enough"
Test Automation

• You don't have to automate everything. Don't try.
• You will have automation bugs.
• Test Automation does not have to be of higher quality than product!
• Quality =
  Reliability + Features + Timliness
• Don't try to make it perfect.

Test Failure Triage

• Not every failure will be a bug.
• Sometimes many failures will all be due to one bug.
• Get used to not pursuing every failure.
• Which is better?
  • Automating 100 testcases with 99% accuracy
  • Automating 200 testcases with 80% accuracy

Benefits of Test Automation

• Allows more testing to happen.
• Tightens test cycle.
• Testing is consistent, repeatable.
• Facilitates additional point/patch releases.
• Makes configuration testing easier.
• Test battery can be continuously improved.

False Benefits

Common hopes that won't pan out

• Fewer testers will be needed.
• It will be easier if it is automated.
• Compensate for poor design, unrealistic schedule.
• No more manual testing.
• Will get your testing organized, documented.

Key Ideas

• Test Automation is Software Development
• Choose Carefully What to Automate
• Use a Test Driver Architecture
• Use an Error Recovery System
• Avoid False Positives
• Accept "Good Enough" Test Automation

It's OK not to automate.

If you are going to do it,

do it right.

Bret Pettichord

BMC Software

bret_pettichord@bmc.com

512-795-6019