June 22, 2008
WatirCraft is Betting on Test Automation
Pete Dignan and I are founding a company called WatirCraft. WatirCraft is a for-profit company that is supporting and maintaining Watir and providing additional products and services based on Watir. We are betting that we can build a business around making testers successful with automated testing.
Our top priorities right now are providing ongoing support for Watir as well as improving Watir’s ability to operate with more browsers and platforms. We recently released an updated version of Watir that included a number of patches contributed by the community, as well as a performance improvement.
Watir’s number one weakness today is its focus on Windows and IE. We are working closely with Angrez Singh and the FireWatir team to bring our code bases together, rejoin the projects and improve the compatability between our implentations. Testers have every right to expect tests they write for one browser to able to be executed with another— without the use of browser-specific logic.
FireWatir supports Firefox on Windows, Linux and Mac. We are also looking further. We have recently had serious inquiries from test architects working on the development teams of two other important browsers, seeking advice on how to make Watir work with their browsers. And we also have plans to integrate Watir with Selenium-RC.
Many people today think of Watir as something that works with Windows and Internet Explorer. We seek to change that. We think Watir’s defining characteristic is its intuitive API for operating a browser. This API is sufficiently appealing that it has been translated to C# (Watin) and Java (Watij). And of course there are a couple of implementations of this API in Ruby (Watir, FireWatir, SafariWatir, Celerity). But we need to keep these Ruby implementations in sync, which is why we are developing a browser-independent test suite to validate Watir compatability between these code bases.
Making Watir more portable and more supportable will mean changes, some of which will be disruptive. We will be making some features more robust, and deprecating others. We plan to continue to support Watir as a community project. In fact, the Watir community is the key asset that we are building our business on. Every day, Watir users are helping each other, sharing their work and their visions, and making it easier for the next person who comes along. Pete, new to the Watir community, is amazed at the patience, good faith and dedication he has seen.
By founding this business, Pete and I are making a bet on Watir— not just the tool as it is today, but also the vision that inspires it. It is a vision that realizes that automation is about code and success requires understanding code, and making code understandable.
We are also betting that people want and will be willing to pay for a framework that is easier to use.
It has been great to learn about all the frameworks that have been built for Watir. Watir itself was always conceived as a library that could be used with any framework. Originally we mostly used Test::Unit, which was convenient and well-supported. But with time, other unit testing frameworks (such as Rspec and Shoulda) have arisen and it is just as easy to use Watir with them. It has been great to learn from all the frameworks that Watir users have created and shared.
Watir was concieved as much as an embodied critique of the then-prevailing testing tools as it was as a foundation for successful automation. It was thus designed around a standard scripting language, not a vendorscript. It was designed to be easy to write, rather than easy to record. And it was designed to be a library and not a framework. Many of the commercial tools have been burdened by tight bindings to leaden frameworks.
WatirCraft is developing a framework that will allow testers who aren’t wizards (and who don’t have wizards around to help) to nonetheless hit the ground running with Watir. Our framework won’t be for everybody and we will continue to encourage Watir’s many users to use whatever framework they wish, whether it be open-source, home-brewed or ours. Our goal, really, is to make Watir-based testing accessible to more people.
We are also betting on Ruby. We still sometimes hear regrets from testers that Watir isn’t written in Python. I don’t regret it one bit. I am thankful that Brian Marick pulled me on to the Ruby train with him years ago. I believe that we can build a better framework with less effort in Ruby than in Python. This is because Ruby has a set of features called meta-programming.
The advantages of meta-programming have been hard to explain to testers. “Meta-programming” is actually a new label for a set of advanced language features that Ruby has had for a long time. But these features are hard to explain and understand and most testers won’t be using them. Nonetheless, Ruby’s meta-programming features make it easier to build consise, intuitive frameworks. Indeed, this is a big reason behind the success of Rails.
We are also betting that the growing use of Agile Development practices will continue. Testers on agile teams are being hammered by shorter iterations. Regression testing is more and more necessary and yet there is less and less time to do it. At the same time, Agile Development is getting testers and developers to work together more closely—exactly what needs to happen if successful test automation is going to happen. We will continue to support automated testing that works not just for testers but for the whole team, and that integrates with agile practices such as developer testing, continuous integration and frequent commits. Personally, I’ve been working exclusively with Agile teams over the past four years. Only when you run tests daily (or more frequently) will you really do what is necessary to make automated testing work.
Today, most of the innovation in automated testing today is coming from agile developers. I find it quite exciting and hard to keep up with, there is so much of it. Yet many of these tools are written for developers, and testers have trouble using them. Indeed developers in some companies are already taking over responsibility for automating testing. Will this be the trend? Will there be a role for testers in the future of automated testing? Will this role be defined by developers?
We think testers do have a significant role in the future of automated testing. By working closely with both testers and developers we can make Watir an even better tool and sell additional tools and support that will make testers successful in an increasingly Agile world. We’re betting on it.
June 16, 2008
Watir 1.5.6 released with "Zippy" speed
Last week we released a new version of Watir. We've mostly been accumulating bug fixes and minor enhancements. However there is one new feature that may be worth the upgrade. We've added a speed improvement to Watir, which we call "Zippy" speed.Installation
Watir now has a two-step install process (assuming you've already installed Ruby). Type this at the command prompt:gem update --system
gem install watirWe are also now recommending that people use Ruby 1.8.6-26 final with Watir. A Ruby bug affecting many Watir users in earlier versions of 1.8.6 is now fixed. If you are upgrading your version of Ruby, you basically have to do a fresh install. There is no upgrade option. Sorry.
Zippy Speed
When you first use Watir, it is rather slow. This is intentional. We made Watir run slow at first because we found that new users wanted to be able to watch their scripts and make sure they were working correctly. So we added some delays to slow Watir down to an auditable speed.However, once you have confidence in your tests, you probably want to run them faster. This is just one command.
ie = Watir::IE.new
ie.speed = :fastWhat is new is that we now have an even faster way to run Watir. We call this "zippy" and it works the same way.
ie = Watir::IE.new
ie.speed = :zippyThe new :zippy speed speeds up tests by enter text fields all at once. If you have tests that require character-by-character text entry, you'll need to continue to use :fast.
In the past, many Watir users realized that they could speed up their tests by using value instead of set. Thus, instead of
ie.text_field(:id, 'payment').set '25'ie.text_field(:id, 'payment').value = '25'Zippy speed was originally included in Watir 1.5.4, but I've been reluctant to brag about it until we got our install problems fixed. It is based on a proof of concept by Jonathan Kohl.
Should You Upgrade?
Other features new since 1.5.3 (released last fall) include support for Chinese character input and the ability to locate more elements using multiple attributes. However, multiple attribute support for text fields, buttons and other input elements has yet to be implemented. We know you are waiting for it.You can now call the row method from within a table, also frequently requested. There are a number of other fixes. (See the Release Notes for details.)
I know many of you are still using Watir 1.4.1. There is only one reason I know of why you should not upgrade. We have had several reports that Watir 1.5.3 and later is slower with some complex pages. We recently got a reproducible case of this problem and will be looking at fixing it in 1.5.7.
June 09, 2008
The Next Advance in Testing
Ward Cunningham says:I suspect there will be a tool at the center of the next advance in testing. The tool will enable the advance, but it won't be the advance. Like with jUnit, the tool will be useful before the advance has happened, but, when the books are written, they will devote far more attention to the practices surrounding the tool than the tool itself.
May 19, 2008
Next Watir Austin Meeting: Jim Matthews' Link Checker
The next meeting of the Austin Watir Users Group will be this Wednesday, May 21.
Jim Matthews will be speaking about his link checker, a web crawler optimized for testing that he has released as open-source. He needed a script that would crawl a website looking for certain special conditions. He built a tool using Ruby, Watir, and Mysql that he has been using to test the website at Callaway Golf. Jim will discuss the problems encountered and the evolution to the tool.
I met Jim years ago during a consulting assignment at Tonic, where he had built his own browser tester in VB. I later developed Watir partly inspired by Jim’s work. Later we worked together at DataCert, where we both used Watir to test different products.
We are meeting at Uplogix again. Try to get to the meeting by 6:30 if you can. We’ll have pizza for you. Complete driving directions.
Update: Use the following Subversion command to check out the link-checker:
svn checkout svn://rubyforge.org/var/svn/test-system/link_checker/trunk
April 30, 2008
What I've Been Using Lately
On my last project we used Watir 1.5.4 (only recently publicly released, but we'd been using it since last fall) and Ruby 1.8.5. There are several problems with Ruby 1.8.6 and therefore I will not use it.
We used Rspec on this last project, and I really liked it. We started with Rspec 0.8, and at that time we looked at the Story Runner, but it clearly wasn't ready for production testing. We later upgraded to Rspec 1.1.1. but did not have time to revisit using the Story Runner. The code seems solid now, and I'm eager to try it out on a real project.
We used Eclipse as our IDE, version 3.3 with the Aptana plugin. There is a new version of Aptana that I plan to install (standalone) when I get a moment. Our developers used IntelliJ and really liked it. I was able to get a comp license for it, but haven't actually installed it yet. I think I'm reluctant to use a tool that might be beyond the reach of our users.
April 29, 2008
Reintegrating Watir and FireWatir
Right now the top priority of the Watir project is to reintegrate with FireWatir and improve the compatability between the two implementations. I've begun serious discussions with Angrez Singh, the FireWatir lead, and we've agreed on this basic objective. There are, however, lots of logistical and architectural issues that need to be discussed further. We've agreed to continue this discussion on the Watir Development list, wtr-development@rubyforge.org.If you would like to be part of this discussion, or learn more about our plans, please join this list. This is a public list that has been quiet lately, but has been used for similar discussions in the past.
April 22, 2008
Rasta supports data-driven testing
Rasta is a data-driven testing framework that pulls data out of Excel spreadsheets and executes it using simple "fixtures" that you write in Ruby. Your fixture methods could use Watir or another Ruby-based library (Selenium-RC, Mechanize, Soap4R) to execute against the target application.
There are several things that I think are simply terrific about Rasta. The most dramatic feature is that the spreadsheet is annotated in real-time as the tests execute. Each cell is colored red or green to indicate whether that step passed. If there are errors, they are attached to the cell as a comment that appears when you hover over it.
There are a lot of data-driven testing frameworks that work off of tables these days. Most use wiki's to create the tables, which feels somewhat barbaric. Rasta allows you to use a spreadsheet instead.
Another great thing about Rasta is that it allows test cases to be defined either horizontally or vertically. I've noticed that most data-driven frameworks (Fit, TestFrame, Certify, QTP) specify test cases horizontally. However, most testers, if they are starting from scratch will organize their test matrices vertically. I'm sure there is some psychological reason for this. In any case, Rasta allows you to organize your tests either way. To me this is simply a great of example of how test tools should be adapted to the way tester's think rather than the other way around.
Another feature along these lines is the ability to stop the tests and then pick up where they left off, instead of having to restart from the beginning.
Right now, Rasta only supports Excel, but the developers are working on adding support for Open Office and Google spreadsheet. They've put together a great website, with screen shots and examples. It really gives a much better impression of what the tool is like than I can convey here.
There has been a lot of discussion in the Watir community about data-driven testing frameworks. Any one interested really should give Rasta a look. It is simple, effective, and fun.
April 12, 2008
Supporting Watir
Pete Dignan and I are founding a company to support Watir and other open-source testing tools. We are still working on the business plan and name of the business. Normally, I would have wanted to wait until we had a few more details decided before making an announcement, but our plans were leaked out early so there is really no point in keeping quiet.
Pete is the president of ProtoTest, a Denver-based testing services company, with both financial success and an excellent track record of client satisfaction. One of the reasons that I’ve agreed to join Pete in this venture is because of his strong commitment to open-source software and continuing to respect and support the Watir community.
One of the first things we’ll be doing is to get a new release of Watir out the door. This has been overdue; me and the other core committers have been very busy over the past six months with other work. The most important thing that we’ll be working on is making Watir work better with Firefox. Specifically, we would like to merge the FireWatir code back into the original Watir code base. I still think that the best way forward on this project is to instrument our test suite so that it can be run against either project. This means some intermediate work on the test suite, as well as putting the Watir project itself under continuous integration using CruiseControl.rb.
March 17, 2008
Austin Watir Users Group, Weds Mar 19, Keyword Testing with Rasta
I am very please to announce that the first meeting of the Austin Watir Users Group will be this Wednesday, March 19. I've been working for some time with several people to make this happen.
Keyword Driven Testing With Rasta, Hugh McGowan, Convio
Hugh McGowan is presenting his work on Rasta a framework for building keyword-driven tests using Ruby and Excel. This open-source framework fills an important gap for the many testers who are looking for a way to easily put their Watir tests into spreadsheets.
Sponsors
TestCo is providing food and Uplogix is providing beverages. If you are planning to attend, please RSVP to Brent Lavelle so we can have an accurate head count when we order the pizza.
Agenda
6:15 Pizza
7:00 Rasta, Hugh McGowan
Location
Uplogix, Inc.
7600B N. Capital of Texas Hwy.Suite 220
Austin, TX 78731-1189
This is located on 360 between 2222 and Spicewood Springs Road.
Uplogix is next to the apartment complex with the big American flag so if you are coming from the south you will need to make a U-turn on 360 at this complex and get over to the right lane fast. Traffic on 360 from 5 till 7 will be bad.
Austin Watir Users Group
We will be planning additional meetings of the Austin Watir Users Group. To recieve future announcements, please subscribe to
watir-austin@yahoogroups.com
March 10, 2008
Looking for Chief Architect and Senior Rails Developers
Due to recent turnover, my team has senior development leadership positions open. These are great opportunities for people who are committed to Agile development and Ruby on Rails.
We are developing a new product line, mostly following XP, but also influenced by Crystal and Lean. We use test-first development, continuous integration, and automated acceptance testing.
The chief architect needs to define and promote a technical vision for our product line, effectively balancing competing concerns such as customizability and upgradeability. We are looking for someone who knows how to work with an Agile team, developing the architecture incrementally, and work with our business stakeholders, understanding the technical requirements and promoting their technical vision.
The Rails developers we are hiring need to be well-versed in test-driven development and related practices of design patterns and object-oriented design.
These are great opportunities to work with a high-performance team on a challenging project. Contact us if you are interested.
Update: Actually these weren't such great opportunities after all. The project was cancelled and the team was let go.
January 11, 2008
How Big Should a Story Be?
Our recent planning sessions have demonstrated two conflicting forces on story size. The need of customers to see business value and high-level plans pushes towards larger stories, yet the need for developers to be able to complete stories in a single iteration pushes for smaller sizes. I have researched a number of sources to understand more how Agile teams handle this conflict.
The most direct treatment of this issue is by Hai Ton, who in his experience report from the Agile 2007 conference says:
What would your Analyst team do when torn between meeting Customer versus Developer demands? When their needs conflict with one another, how do you appease them both? The report outlines the successful strategy one team used to decompose their Stories to serve both the development team needs to demonstrate progress as well as address Customer needs to see value.
His report describes that they used epics, stories and chapters. Epics are described in the form "As a ..., I would like ... so that ...". Customers were asked to sort epics into "must have", "should have", "could have" and "won't have" (MoSCoW). Epics were broken down into multiple stories.
Stories were individually written to be "independent", "negotiable", "valuable to customers", "estimable", "small" and "testable" (INVEST). Stories were described in the "Given ..., When... Then... " format.
Chapters were used when a story was too big for an iteration, and yet breaking it into two stories would no longer clearly demonstrate customer value. Their solution was to create chapters, and then to delay demonstrating the development work to customers until all the chapters of a story were completed.
There is a lot more good detail in this report (A Strategy for Balancing Business Value and Story Size, Proceedings of the Agile 2007 Conference, Hai Ton, ThoughtWorks), which unfortunately is not freely available online.
Here are some other discussions related to this topic that I found interesting.
- Stories have been getting smaller with time, Jeff Patton
- Consider making multiple passes at a story, to allow for feedback, Alistair Cockburn and Jeff Patton
- Group stories into themes, Dave Larabee
- Timebox the estimating process to allow large backlogs to be estimated (with timeclock), Aslak Hellesoy
A related idea is a Minimum Marketable Feature. This term comes from Software by Numbers and describes a feature with clear business value that can then be broken down into stories in development. I haven’t read this book, but have been getting recommendations from so many people that I clearly need to get my hands on a copy.
