Testing Hotlist Update

Watir Released with Improved Support for Firefox

Last week we released Watir 1.6.2. This is a big release with a lot of new features and fixes, but the focus has been integrating the Watir and FireWatir code bases and improving our support for Firefox. Earlier this year we put both projects under the same umbrella. This release significantly improves compatibility. Many of our users are finding that they can now run all or nearly all of their tests on both IE and Firefox.

We've also provided other features and fixes that will be of interest to users who don't care about Firefox (if such people exist). It is backwards compatabible, so we expect all Watir users will be upgrading to this release. Install or upgrade to this release.

And the end of this note, we mention a few issues that people have encountered when upgrading to this version.

Firefox Support

We've fixed many of the reported compatibility issues. FireWatir now supports nearly all off of the elements supported by Watir. Also both implementations now support being able to specify page elements using multiple attributes (e.g. browser.button(:name => 'Purchase", :index => 2).click). Lots of other issues, detailed in the release notes, have also been fixed.

Both Watir and FireWatir users can simply upgrade and continue to run their existing tests. But they can also make use of our new generic browser interface.

require 'watir'
browser = Watir::Browser.new

Once you have changed your tests to use this instead of Watir::IE.new, then you can configure which browser will be used at runtime.

Modal Dialog Support

We have updated Watir's modal dialog support to work with Ruby 1.8.6. Previously, users who needed to work with the showModalDialog had to use Ruby 1.8.2, but this restriction is no longer in place. We now recommend that all Watir users use the latest version of Ruby 1.8.6.

Speed Improvements on IE

We made some speed improvements in early version of Watir 1.5 that stopped working with version 1.5.4 - 1.5.6. These speed improvements are now working again. Users who had not upgraded to 1.5.6 because of performance concerns should upgrade to this release of Watir.

We've also added unit tests that will alert us if we happen to break this functionality in the future. Sorry about that.

Multiple Attribute Support for All Page Elements

I mentioned this above, but it bears repeating. Previously, Watir allowed you to access some page elements using multiple attributes, but you could only use a single attribute to access input elements such as buttons or text fields or select lists. Now you can use multiple attributes to access any of these elements. There is no limit to the number of attributes you can use. In other words, multiple attribute support has been improved for both IE and Firefox

Don't use "include Watir" at Toplevel

For some time, we've been telling people to avoid using "include Watir" in their scripts because it had the potential to cause unforeseeable problems. With this release, it causes conflicts with one of the other libraries that Watir uses. If you have been using "include Watir" in your scripts please read our detailed advice on this topic.

Builder Error is Benign

When installing Watir 1.6.2, some users will see this error.

ERROR:  While generating documentation for builder-2.1.2
... MESSAGE:   Unhandled special: Special: type=17, text="<!-- HI -->"
... RDOC args: --ri --op e:/ruby/lib/ruby/gems/1.8/doc/builder-2.1.2/ri --title
Builder -- Easy XML Building --main README --line-numbers --quiet lib CHANGES Ra
kefile README doc/releases/builder-1.2.4.rdoc doc/releases/builder-2.0.0.rdoc do
c/releases/builder-2.1.1.rdoc

It is harmless and can be ignored. All it means is that the documentation for the Builder gem could not be generated. Everything else will work.

Unit Tests Won't Run from Gems

You can no longer run Watir's unit tests from within a gem. This is because we have started integrating our Watir/FireWatir test suites and they have cross dependencies. They still run fine from source. You and run them yourself. We also run them everyday and publicly post results.

Two Ruby Books for Watir Testers

Many Watir users want to learn more Ruby so that they can build out their frameworks and libraries. When we created Watir, we knew that testers needed an easy-to-use full-featured language for their automation. Ruby allows most testers to quickly write simple Watir scripts. But as their tests grow and they need to add features and organization to their test suites, they are looking for more Ruby techniques. I pulled together several recommendations for ruby books some time ago. Today, I'd like to focus on the two books that I think Watir testers would benefit from the most.

After teaching Watir/Ruby scripting to dozens of testers, I've found that testers -- even those who insist they are not developers -- can learn effective programming techniques and concepts. But they need to be presented in concrete, practical contexts. Most programming manuals, on the other hand, use artificial examples assume an understanding of abstract concepts, such as exception handling or object-oriented programming. That's why they don't work for many testers.

If you are tester who knows some programming -- but not object-oriented -- and strongly prefers concrete, practical examples, then these recommendations are for you.

Testers looking to better organize their code and better understand how to use classes, modules and libraries will benefit from studying Everyday Scripting with Ruby by Brian Marick. It has a tutorial format, walking you through a series of increasingly richer examples. Previously, I said:

In Everyday Scripting with Ruby, Marick teaches the Ruby language by working through a series of practical scripts. These scripts accomplish everyday tasks that every tester can relate to: comparing file lists, monitoring code changes, scraping web pages, and using a notification framework. The book covers not only the Ruby language, including an excellent chapter on regular expressions, but also provides detailed advice on how to develop reusable libraries, something that Watir users are always asking for help with. If you are a serious Watir user, don’t hesitate to get this book.

Although this book only touches on Watir explicitly for a few pages, it actually has a deep connection. Back in 2002, Marick and I started teaching a class together called "Scripting for Testers". This class was based on our mutual conviction that testers needed to learn the fundamentals of scripting. Watir was one spin off from this class. The other is this book. Watir grew from a small library we used in the class to show how to use scripting to automate a browser. And Marick poured his commitment to teaching testers scripting into this book. As such, the book doesn't just teach Ruby, but rather is really focussed on teaching good development practices, such as when to use global variables and how to do test-driven development.

Marick's book has a tutorial format suitable for serious study. Another good book for testers is the Ruby Cookbook, By Carlson and Richardson. It consists of over 300 recipes, each only a few pages, that can be read in any order. Here's a quick selection of items from the table of contents:

1.19  Validating an Email Address
3.7   Converting Between Time Zones
4.9   Sorting an Array by Frequency of Appearance
5.11  Choosing Randomly from a Weighted List
9.6   Automatically Loading Libraries as Needed
11.13 Extracting All the URLs from an HTML Document
14.5  Sending Mail

Watir Status Update

I've been busy lately working on the next release of Watir, coming out later this month. It will be a combined Watir/FireWatir release. I discuss this work in detail in this podcast. Our work on supporting Firefox is drawing contributors from the community. This prompted me to write up guidelines on making contributions.

At WatirCraft, we are currently looking for customers, particularly those interested in a Watir-based framework.

One of the reasons that I've been posting here less is that I've been posting on twitter and on something I call my miniblog. I've also been frequenting the #watir IRC channel.

Agile is All About Feedback

Agile methods allow your team to get feedback regarding the software you are building. That’s the point. The feedback works on several levels. Pair programming gives developers instant feedback on their code. Stories represent units of work where testers and analysts can give feedback to developers. Iteration releases facilitate feedback from outside the team. Most agile practices are valuable because they create feedback loops that allow teams to adapt.

A lot of teams adopt Agile with a grab-bag approach without quite realizing the point of the practices. They pair-program without discussion or changing drivers. They send code to QA that the testers can’t test because the story boundaries are arbitrary; they can’t tell whether they found a bug or just the end of the story. Iterations become schedule milestones rather than real opportunities to improve alignment and adjust objectives.

The reason Agile teams can do with less planning is because feedback allows you to make sure that you are on course. If you don’t have meaningful feedback then you’re not agile. You’re just in a new form of chaos.

On my last project, we defined our stories so that they made sense to everyone on the team. Our analysts, testers and developers could all understand and review individual stories. But we found that we had to create a larger grouping, which we called features, to facilitate meaningful review from outside our team. We made sure all the stories in a feature were complete before soliciting feedback from outside the team.

Being able to give and receive meaningful feedback is often a challenge for people. Yet it is crucial to success with Agile.

Agile teams get into terrible binds when executives or clients hand them a list of requirements at the start, tell them to use Agile (because its faster), and then don’t want to participate in the feedback process.

Agile isn’t faster all by itself. Agile is only a benefit in a world that acknowledges the value of adapting. And that adaptability needs to go all the way to whoever is funding the project. It is not enough for the team to be Agile. The sponsors need to be Agile too. Are all of the requirements really required? Do we know exactly what the software needs to look like from the start?

Agile is faster because feedback allows you to find and focus on the most valuable features. If you are certain you know what needs to be built, don’t use Agile. If you don’t have time to gather and act on feedback from customers, then don’t use Agile. If you are sure that everyone understands exactly what needs to be done from the start, then don’t use Agile.

Agile practices build a technical and organizational infrastructure to facilitate getting and acting on feedback. If you aren’t going to adapt to feedback, then this infrastracture is waste that will only slow you down. Previous: Lessons Learned in Agile Development Next: Agile Mostly Means Something Between Scrum and XP

FireWatir 1.2.1 is Released

We have a released a new version of FireWatir. This includes some fixes related to Firefox 3. It also marks the merger of the FireWatir and Watir projects. Unlike Watir, FireWatir works on Windows, Mac and Linux.

To install:

1. "gem install firewatir"
2. Install the plugin for Firefox.

Lessons Learned in Agile Development

This begins a series of short essays on Agile development. I am writing these for several reasons.

We are using many Agile methods in the development of Watir. We are using such methods as continuous integration and emergent architecture in our development. Many Watir users are curious about our development, but unfamiliar with some of our methods. I want them to understand.

Many Watir users and other testers are trying to understand Agile. Many are on teams that have recently begun using Agile methods, and are wondering whether it works. It does, but there are also lots of ways to fail. Testers, always quick to find faults, have a hard time with much of the Agile literature, written as it is with developers or managers in mind. I’m always thinking about the testers.

WatirCraft is also using Agile methods. Explaining Agile gives me a chance to explain to the current and future WatirCraft team some of the rationale behind our methods.

I began studying Agile testing six years ago and for the past four years have worked exclusively with talented, high-performing Agile teams. In the past few years, I’ve also been coaching my collegues on Agile methods in areas beyond testing. In that time, I’ve compiled quite a pile of notes on the topic, including key ideas and common misunderstandings. I’m happy to have this chance to share them.

Next: Agile is all about feedback.

Private Blogs

Yesterday, I spent yet another evening researching private blogging options. Several months ago, I started a private blog to share with the people who are helping us launch WatirCraft. I've had these before at previous companies. I think I was the first to start an internal blog at ThoughtWorks. At Dovetail, many people already had them when i arrived. Some used it as a training ground to get more comfortable with the format before blogging publicly. I liked it as a way to publish essays in a way that encouraged a public discussion. Something similar can be done with email, but there are a few drawbacks.

Invariably, email discussions leave someone important out, and can be difficult to add people to the discussion later. Internal blogs don't have this problem: any one in the company who finds out about the discussion can join in, sometimes even months later. I also find myself reluctant to send out emails because they can be so interrupting. With a blog, it seems easier to write something that is important but not urgent, and you'll get more thoughtful replies because people know that they replies will be archived.

I remember pushing the idea that QA people should be called “Testers” (and not “QA”) on my ThoughtWorks blog. We had pretty much come to agree on this point in the Context-Driven community before I joined ThoughtWorks, but I came to see that ThoughtWorks, like the industry at large, still labelled the role “QA”. Because I was then in a position of some standing (Director of Testing Practice), making an issue of this on my public blog would have been a positional move, likely polarizing both me and those who disagreed. By publishing my position internally, I essentially acknowledged that I wanted feedback from my collegues on this topic. I remember Rebecca Parsons suggested “QA/Tester,” which, though awkward, didn't run into the various objections I had laid out against QA, and since then I've been more comfortable with the “QA” term, even though I still usually prefer “Testers”.

At both ThoughtWorks and Dovetail, we made our blogs private simply by publishing regular blogs on our internal networks. You either had to be in our offices or connect over the VPN to see them. Web-based feedreaders (like Google Reader) wouldn't work, but you could use any client-based feedreader (Outlook has one) when you were connected to the company network.

I looked closely at Blogger and WordPress and also more generally across the field and was not able to find any ready-to-use blog engine that supported private blogs and provided a feed. It is possible for feeds to be password protected. For example, we use Highrise and it provides a password-protected feed containing recent changes. But i could not find such a thing for any blogs.

Apparently several blog engines added support for private blogging a couple of years ago, but left public feeds on them. These were crawled by search engines and people were finding their private posts on Google! The quick fix was to remove the feeds, and apparently the issue hasn't been addressed since. One option that might work for some would be to have a blog that isn't strictly private, but is protected from search engines, so you'd have to know the URL to read it. This option is offered by WordPress.

At WatirCraft, we have not yet set up a VPN, so that is not an option for us. For now, it seems the best way to run an internal blog is to make one private and then to set up an accompanying mailing list that you post to when there are updates on the blog.

Setting up Watir with Cruise Control

Last week I set up Watir so that its unit tests are run automatically whenever any changes are committed (“checked in”) to the development repository. This is done with Cruise Control, a tool which watches repositories, runs tests and posts results. The Watir tests are run at watirbuild.com and you can check the latest test results yourself. Indeed, the page has a button called “build now.” Click it and the tests will be run immediately on the server.

The general term for this process is continuous integration—a common practice with Agile teams. I’ll write more about why this practice is important for the Watir project another day. Today’s post is about what I did to set this up.

First, I got a server, a virtual private server (VPS) running Windows from VPSLAND. I would have preferred a machine running XP, since most Watir users use that, but the closest I could get was 32-bit Windows 2003 Server. Cost: $19 a month, courtesy of WatirCraft. This VPS is named watirbuild.com.

Next I installed the Google Toolbar, because I can’t find anything without it. I also had to loosen up security on IE so that I could use it to download the software I needed. (The server VPSland provided came with security really locked down.)

I then installed Ruby 1.8.6-26. The next step was to do a “gem update --system” to update the gem installer to the latest version (which Watir needs), but this command failed with this error.

c:/ruby/1.8/yaml.rb:133: [BUG] Segmentation fault

A little research showed that this was a common problem on RAM-constrained VPS’s. Apparently the pre-update gem installer is a memory hog and the VPS’s don’t have any swap space. The common workaround has been manually download the gems before installing them. (Only the remote gem installer runs into memory problems.) But I used a different approach.

I wanted our test environment to match, as much as possible, the environments that other people are using for testing, so I really wanted to have the latest gem installer. I also knew that it had better memory handling. So what I did was install Ruby on one of my own systems with more RAM, and then did a “gem update --system” on it. Then I zipped up the resulting “ruby” directory (using 7zip), uploaded it to watirbuild.com and then unzipped it there over the original installation. I was then able to do remote gem installs without any further trouble.

Because Watir now depends on several other gems. I did a “gem install watir” and then uninstalled the Watir gem itself. This left all the dependent gems installed.

Watir uses Subversion (SVN) as its development repository, so we needed a client to access that code. I installed the command line client.

The next step was to install the continuous integration tool itself. I chose CruiseControl.rb because it is Ruby-based and easy to configure. Pre-deployment testing had shown, however, that the latest release (1.3.0) did not work on Windows (After starting the server, it would get repeated "Access is denied." errors from SVN.) Instead I needed to use the latest from its development repository. They recently migrated that project from Subversion to Git, so that meant I needed to install a Git client. Once it was installed, installing Cruise was simply a matter of doing

git clone git://rubyforge.org/cruisecontrolrb.git

With Cruise installed, I added the Watir project:

cruise add watir https://svn.openqa.org/svn/watir/trunk/watir

I also needed to add a “cruise” target to Watir’s rakefile. Which I did with the following code:

desc 'Run unit tests'
task :test do
  load 'unittests/core_tests.rb'
end

task :cruise => :test

(This code is now part of the Watir project, so you wouldn't need to provide this code if you were running Watir. But you will need something similar if you are setting up your own project with CruiseControl.rb.)

Then I started Cruise using “cruise start -p 80”, and this immediately kicked off Watir’s unit tests. Some failed on first run, and I had to modify a couple of IE’s security settings to fix this.

To test everything, I checked in a minor documentation change to Watir and verified that the unit tests were automatically kicked off.

WatirCraft is Betting on Test Automation

Pete Dignan and I are founding a company called WatirCraft. WatirCraft is a for-profit company that is supporting and maintaining Watir and providing additional products and services based on Watir. We are betting that we can build a business around making testers successful with automated testing.

Our top priorities right now are providing ongoing support for Watir as well as improving Watir’s ability to operate with more browsers and platforms. We recently released an updated version of Watir that included a number of patches contributed by the community, as well as a performance improvement.

Watir’s number one weakness today is its focus on Windows and IE. We are working closely with Angrez Singh and the FireWatir team to bring our code bases together, rejoin the projects and improve the compatability between our implentations. Testers have every right to expect tests they write for one browser to able to be executed with another— without the use of browser-specific logic.

FireWatir supports Firefox on Windows, Linux and Mac. We are also looking further. We have recently had serious inquiries from test architects working on the development teams of two other important browsers, seeking advice on how to make Watir work with their browsers. And we also have plans to integrate Watir with Selenium-RC.

Many people today think of Watir as something that works with Windows and Internet Explorer. We seek to change that. We think Watir’s defining characteristic is its intuitive API for operating a browser. This API is sufficiently appealing that it has been translated to C# (Watin) and Java (Watij). And of course there are a couple of implementations of this API in Ruby (Watir, FireWatir, SafariWatir, Celerity). But we need to keep these Ruby implementations in sync, which is why we are developing a browser-independent test suite to validate Watir compatability between these code bases.

Making Watir more portable and more supportable will mean changes, some of which will be disruptive. We will be making some features more robust, and deprecating others. We plan to continue to support Watir as a community project. In fact, the Watir community is the key asset that we are building our business on. Every day, Watir users are helping each other, sharing their work and their visions, and making it easier for the next person who comes along. Pete, new to the Watir community, is amazed at the patience, good faith and dedication he has seen.

By founding this business, Pete and I are making a bet on Watir— not just the tool as it is today, but also the vision that inspires it. It is a vision that realizes that automation is about code and success requires understanding code, and making code understandable.

We are also betting that people want and will be willing to pay for a framework that is easier to use.

It has been great to learn about all the frameworks that have been built for Watir. Watir itself was always conceived as a library that could be used with any framework. Originally we mostly used Test::Unit, which was convenient and well-supported. But with time, other unit testing frameworks (such as Rspec and Shoulda) have arisen and it is just as easy to use Watir with them. It has been great to learn from all the frameworks that Watir users have created and shared.

Watir was concieved as much as an embodied critique of the then-prevailing testing tools as it was as a foundation for successful automation. It was thus designed around a standard scripting language, not a vendorscript. It was designed to be easy to write, rather than easy to record. And it was designed to be a library and not a framework. Many of the commercial tools have been burdened by tight bindings to leaden frameworks.

WatirCraft is developing a framework that will allow testers who aren’t wizards (and who don’t have wizards around to help) to nonetheless hit the ground running with Watir. Our framework won’t be for everybody and we will continue to encourage Watir’s many users to use whatever framework they wish, whether it be open-source, home-brewed or ours. Our goal, really, is to make Watir-based testing accessible to more people.

We are also betting on Ruby. We still sometimes hear regrets from testers that Watir isn’t written in Python. I don’t regret it one bit. I am thankful that Brian Marick pulled me on to the Ruby train with him years ago. I believe that we can build a better framework with less effort in Ruby than in Python. This is because Ruby has a set of features called meta-programming.

The advantages of meta-programming have been hard to explain to testers. “Meta-programming” is actually a new label for a set of advanced language features that Ruby has had for a long time. But these features are hard to explain and understand and most testers won’t be using them. Nonetheless, Ruby’s meta-programming features make it easier to build consise, intuitive frameworks. Indeed, this is a big reason behind the success of Rails.

We are also betting that the growing use of Agile Development practices will continue. Testers on agile teams are being hammered by shorter iterations. Regression testing is more and more necessary and yet there is less and less time to do it. At the same time, Agile Development is getting testers and developers to work together more closely—exactly what needs to happen if successful test automation is going to happen. We will continue to support automated testing that works not just for testers but for the whole team, and that integrates with agile practices such as developer testing, continuous integration and frequent commits. Personally, I’ve been working exclusively with Agile teams over the past four years. Only when you run tests daily (or more frequently) will you really do what is necessary to make automated testing work.

Today, most of the innovation in automated testing today is coming from agile developers. I find it quite exciting and hard to keep up with, there is so much of it. Yet many of these tools are written for developers, and testers have trouble using them. Indeed developers in some companies are already taking over responsibility for automating testing. Will this be the trend? Will there be a role for testers in the future of automated testing? Will this role be defined by developers?

We think testers do have a significant role in the future of automated testing. By working closely with both testers and developers we can make Watir an even better tool and sell additional tools and support that will make testers successful in an increasingly Agile world. We’re betting on it.

Watir 1.5.6 released with "Zippy" speed

Last week we released a new version of Watir. We've mostly been accumulating bug fixes and minor enhancements. However there is one new feature that may be worth the upgrade. We've added a speed improvement to Watir, which we call "Zippy" speed.

Installation

Watir now has a two-step install process (assuming you've already installed Ruby). Type this at the command prompt:

gem update --system
gem install watir

This will update your version of the Rubygems installer and then install the latest version of Watir. Some people have had install problems with 1.5.4 and 1.5.5. Those problems are now fixed.

We are also now recommending that people use Ruby 1.8.6-26 final with Watir. A Ruby bug affecting many Watir users in earlier versions of 1.8.6 is now fixed. If you are upgrading your version of Ruby, you basically have to do a fresh install. There is no upgrade option. Sorry.

Zippy Speed

When you first use Watir, it is rather slow. This is intentional. We made Watir run slow at first because we found that new users wanted to be able to watch their scripts and make sure they were working correctly. So we added some delays to slow Watir down to an auditable speed.

However, once you have confidence in your tests, you probably want to run them faster. This is just one command.

ie = Watir::IE.new
ie.speed = :fast

I know this is a surprise to some Watir users: we haven't done a great job of letting this trick be known. This feature has been in Watir for a long time, certainly as far back as Watir 1.4.

What is new is that we now have an even faster way to run Watir. We call this "zippy" and it works the same way.

ie = Watir::IE.new
ie.speed = :zippy

Normally, when Watir enters text into a text field, it does so letter by letter. Two reasons. One, it looks cooler. Two, there are rare cases where you have to do this to realistically engage Javascript attached to the text field that, say, limits the characters that can be entered. But with tests that had lots of data entry, this could really slow things down.

The new :zippy speed speeds up tests by enter text fields all at once. If you have tests that require character-by-character text entry, you'll need to continue to use :fast.

In the past, many Watir users realized that they could speed up their tests by using value instead of set. Thus, instead of

ie.text_field(:id, 'payment').set '25'

They would use

ie.text_field(:id, 'payment').value = '25'

The new :zippy speed does exactly the same thing, only behind the scenes so you don't have to modify your individual scripts.

Zippy speed was originally included in Watir 1.5.4, but I've been reluctant to brag about it until we got our install problems fixed. It is based on a proof of concept by Jonathan Kohl.

Should You Upgrade?

Other features new since 1.5.3 (released last fall) include support for Chinese character input and the ability to locate more elements using multiple attributes. However, multiple attribute support for text fields, buttons and other input elements has yet to be implemented. We know you are waiting for it.

You can now call the row method from within a table, also frequently requested. There are a number of other fixes. (See the Release Notes for details.)

I know many of you are still using Watir 1.4.1. There is only one reason I know of why you should not upgrade. We have had several reports that Watir 1.5.3 and later is slower with some complex pages. We recently got a reproducible case of this problem and will be looking at fixing it in 1.5.7.

The Next Advance in Testing

Ward Cunningham says:

I suspect there will be a tool at the center of the next advance in testing. The tool will enable the advance, but it won't be the advance. Like with jUnit, the tool will be useful before the advance has happened, but, when the books are written, they will devote far more attention to the practices surrounding the tool than the tool itself.