The general term for this process is continuous integration—a common practice with Agile teams. I’ll write more about why this practice is important for the Watir project another day. Today’s post is about what I did to set this up.
First, I got a server, a virtual private server (VPS) running Windows from VPSLAND. I would have preferred a machine running XP, since most Watir users use that, but the closest I could get was 32-bit Windows 2003 Server. Cost: $19 a month, courtesy of WatirCraft. This VPS is named watirbuild.com.
Next I installed the Google Toolbar, because I can’t find anything without it. I also had to loosen up security on IE so that I could use it to download the software I needed. (The server VPSland provided came with security really locked down.)
I then installed Ruby 1.8.6-26. The next step was to do a “gem update --system” to update the gem installer to the latest version (which Watir needs), but this command failed with this error.
c:/ruby/1.8/yaml.rb:133: [BUG] Segmentation fault
A little research showed that this was a common problem on RAM-constrained VPS’s. Apparently the pre-update gem installer is a memory hog and the VPS’s don’t have any swap space. The common workaround has been manually download the gems before installing them. (Only the remote gem installer runs into memory problems.) But I used a different approach.
I wanted our test environment to match, as much as possible, the environments that other people are using for testing, so I really wanted to have the latest gem installer. I also knew that it had better memory handling. So what I did was install Ruby on one of my own systems with more RAM, and then did a “gem update --system” on it. Then I zipped up the resulting “ruby” directory (using 7zip), uploaded it to watirbuild.com and then unzipped it there over the original installation. I was then able to do remote gem installs without any further trouble.
Because Watir now depends on several other gems. I did a “gem install watir” and then uninstalled the Watir gem itself. This left all the dependent gems installed.
Watir uses Subversion (SVN) as its development repository, so we needed a client to access that code. I installed the command line client.
The next step was to install the continuous integration tool itself. I chose CruiseControl.rb because it is Ruby-based and easy to configure. Pre-deployment testing had shown, however, that the latest release (1.3.0) did not work on Windows (After starting the server, it would get repeated "Access is denied." errors from SVN.) Instead I needed to use the latest from its development repository. They recently migrated that project from Subversion to Git, so that meant I needed to install a Git client. Once it was installed, installing Cruise was simply a matter of doing
git clone git://rubyforge.org/cruisecontrolrb.git
With Cruise installed, I added the Watir project:
cruise add watir https://svn.openqa.org/svn/watir/trunk/watir
I also needed to add a “cruise” target to Watir’s rakefile. Which I did with the following code:
desc 'Run unit tests' task :test do load 'unittests/core_tests.rb' end task :cruise => :test
(This code is now part of the Watir project, so you wouldn't need to provide this code if you were running Watir. But you will need something similar if you are setting up your own project with CruiseControl.rb.)
Then I started Cruise using “cruise start -p 80”, and this immediately kicked off Watir’s unit tests. Some failed on first run, and I had to modify a couple of IE’s security settings to fix this.
To test everything, I checked in a minor documentation change to Watir and verified that the unit tests were automatically kicked off.
]]>Our top priorities right now are providing ongoing support for Watir as well as improving Watir’s ability to operate with more browsers and platforms. We recently released an updated version of Watir that included a number of patches contributed by the community, as well as a performance improvement.
Watir’s number one weakness today is its focus on Windows and IE. We are working closely with Angrez Singh and the FireWatir team to bring our code bases together, rejoin the projects and improve the compatability between our implentations. Testers have every right to expect tests they write for one browser to able to be executed with another— without the use of browser-specific logic.
FireWatir supports Firefox on Windows, Linux and Mac. We are also looking further. We have recently had serious inquiries from test architects working on the development teams of two other important browsers, seeking advice on how to make Watir work with their browsers. And we also have plans to integrate Watir with Selenium-RC.
Many people today think of Watir as something that works with Windows and Internet Explorer. We seek to change that. We think Watir’s defining characteristic is its intuitive API for operating a browser. This API is sufficiently appealing that it has been translated to C# (Watin) and Java (Watij). And of course there are a couple of implementations of this API in Ruby (Watir, FireWatir, SafariWatir, Celerity). But we need to keep these Ruby implementations in sync, which is why we are developing a browser-independent test suite to validate Watir compatability between these code bases.
Making Watir more portable and more supportable will mean changes, some of which will be disruptive. We will be making some features more robust, and deprecating others. We plan to continue to support Watir as a community project. In fact, the Watir community is the key asset that we are building our business on. Every day, Watir users are helping each other, sharing their work and their visions, and making it easier for the next person who comes along. Pete, new to the Watir community, is amazed at the patience, good faith and dedication he has seen.
By founding this business, Pete and I are making a bet on Watir— not just the tool as it is today, but also the vision that inspires it. It is a vision that realizes that automation is about code and success requires understanding code, and making code understandable.
We are also betting that people want and will be willing to pay for a framework that is easier to use.
It has been great to learn about all the frameworks that have been built for Watir. Watir itself was always conceived as a library that could be used with any framework. Originally we mostly used Test::Unit, which was convenient and well-supported. But with time, other unit testing frameworks (such as Rspec and Shoulda) have arisen and it is just as easy to use Watir with them. It has been great to learn from all the frameworks that Watir users have created and shared.
Watir was concieved as much as an embodied critique of the then-prevailing testing tools as it was as a foundation for successful automation. It was thus designed around a standard scripting language, not a vendorscript. It was designed to be easy to write, rather than easy to record. And it was designed to be a library and not a framework. Many of the commercial tools have been burdened by tight bindings to leaden frameworks.
WatirCraft is developing a framework that will allow testers who aren’t wizards (and who don’t have wizards around to help) to nonetheless hit the ground running with Watir. Our framework won’t be for everybody and we will continue to encourage Watir’s many users to use whatever framework they wish, whether it be open-source, home-brewed or ours. Our goal, really, is to make Watir-based testing accessible to more people.
We are also betting on Ruby. We still sometimes hear regrets from testers that Watir isn’t written in Python. I don’t regret it one bit. I am thankful that Brian Marick pulled me on to the Ruby train with him years ago. I believe that we can build a better framework with less effort in Ruby than in Python. This is because Ruby has a set of features called meta-programming.
The advantages of meta-programming have been hard to explain to testers. “Meta-programming” is actually a new label for a set of advanced language features that Ruby has had for a long time. But these features are hard to explain and understand and most testers won’t be using them. Nonetheless, Ruby’s meta-programming features make it easier to build consise, intuitive frameworks. Indeed, this is a big reason behind the success of Rails.
We are also betting that the growing use of Agile Development practices will continue. Testers on agile teams are being hammered by shorter iterations. Regression testing is more and more necessary and yet there is less and less time to do it. At the same time, Agile Development is getting testers and developers to work together more closely—exactly what needs to happen if successful test automation is going to happen. We will continue to support automated testing that works not just for testers but for the whole team, and that integrates with agile practices such as developer testing, continuous integration and frequent commits. Personally, I’ve been working exclusively with Agile teams over the past four years. Only when you run tests daily (or more frequently) will you really do what is necessary to make automated testing work.
Today, most of the innovation in automated testing today is coming from agile developers. I find it quite exciting and hard to keep up with, there is so much of it. Yet many of these tools are written for developers, and testers have trouble using them. Indeed developers in some companies are already taking over responsibility for automating testing. Will this be the trend? Will there be a role for testers in the future of automated testing? Will this role be defined by developers?
We think testers do have a significant role in the future of automated testing. By working closely with both testers and developers we can make Watir an even better tool and sell additional tools and support that will make testers successful in an increasingly Agile world. We’re betting on it.
]]>gem update --system
gem install watirWe are also now recommending that people use Ruby 1.8.6-26 final with Watir. A Ruby bug affecting many Watir users in earlier versions of 1.8.6 is now fixed. If you are upgrading your version of Ruby, you basically have to do a fresh install. There is no upgrade option. Sorry.
However, once you have confidence in your tests, you probably want to run them faster. This is just one command.
ie = Watir::IE.new
ie.speed = :fastWhat is new is that we now have an even faster way to run Watir. We call this "zippy" and it works the same way.
ie = Watir::IE.new
ie.speed = :zippyThe new :zippy speed speeds up tests by enter text fields all at once. If you have tests that require character-by-character text entry, you'll need to continue to use :fast.
In the past, many Watir users realized that they could speed up their tests by using value instead of set. Thus, instead of
ie.text_field(:id, 'payment').set '25'ie.text_field(:id, 'payment').value = '25'Zippy speed was originally included in Watir 1.5.4, but I've been reluctant to brag about it until we got our install problems fixed. It is based on a proof of concept by Jonathan Kohl.
You can now call the row method from within a table, also frequently requested. There are a number of other fixes. (See the Release Notes for details.)
I know many of you are still using Watir 1.4.1. There is only one reason I know of why you should not upgrade. We have had several reports that Watir 1.5.3 and later is slower with some complex pages. We recently got a reproducible case of this problem and will be looking at fixing it in 1.5.7. ]]>
I suspect there will be a tool at the center of the next advance in testing. The tool will enable the advance, but it won't be the advance. Like with jUnit, the tool will be useful before the advance has happened, but, when the books are written, they will devote far more attention to the practices surrounding the tool than the tool itself.
]]>Jim Matthews will be speaking about his link checker, a web crawler optimized for testing that he has released as open-source. He needed a script that would crawl a website looking for certain special conditions. He built a tool using Ruby, Watir, and Mysql that he has been using to test the website at Callaway Golf. Jim will discuss the problems encountered and the evolution to the tool.
I met Jim years ago during a consulting assignment at Tonic, where he had built his own browser tester in VB. I later developed Watir partly inspired by Jim’s work. Later we worked together at DataCert, where we both used Watir to test different products.
We are meeting at Uplogix again. Try to get to the meeting by 6:30 if you can. We’ll have pizza for you. Complete driving directions.
Update: Use the following Subversion command to check out the link-checker:
svn checkout svn://rubyforge.org/var/svn/test-system/link_checker/trunk]]>
We used Rspec on this last project, and I really liked it. We started with Rspec 0.8, and at that time we looked at the Story Runner, but it clearly wasn't ready for production testing. We later upgraded to Rspec 1.1.1. but did not have time to revisit using the Story Runner. The code seems solid now, and I'm eager to try it out on a real project.
We used Eclipse as our IDE, version 3.3 with the Aptana plugin. There is a new version of Aptana that I plan to install (standalone) when I get a moment. Our developers used IntelliJ and really liked it. I was able to get a comp license for it, but haven't actually installed it yet. I think I'm reluctant to use a tool that might be beyond the reach of our users.
]]>If you would like to be part of this discussion, or learn more about our plans, please join this list. This is a public list that has been quiet lately, but has been used for similar discussions in the past.
]]>There are several things that I think are simply terrific about Rasta. The most dramatic feature is that the spreadsheet is annotated in real-time as the tests execute. Each cell is colored red or green to indicate whether that step passed. If there are errors, they are attached to the cell as a comment that appears when you hover over it.
There are a lot of data-driven testing frameworks that work off of tables these days. Most use wiki's to create the tables, which feels somewhat barbaric. Rasta allows you to use a spreadsheet instead.
Another great thing about Rasta is that it allows test cases to be defined either horizontally or vertically. I've noticed that most data-driven frameworks (Fit, TestFrame, Certify, QTP) specify test cases horizontally. However, most testers, if they are starting from scratch will organize their test matrices vertically. I'm sure there is some psychological reason for this. In any case, Rasta allows you to organize your tests either way. To me this is simply a great of example of how test tools should be adapted to the way tester's think rather than the other way around.
Another feature along these lines is the ability to stop the tests and then pick up where they left off, instead of having to restart from the beginning.
Right now, Rasta only supports Excel, but the developers are working on adding support for Open Office and Google spreadsheet. They've put together a great website, with screen shots and examples. It really gives a much better impression of what the tool is like than I can convey here.
There has been a lot of discussion in the Watir community about data-driven testing frameworks. Any one interested really should give Rasta a look. It is simple, effective, and fun.
]]>Pete is the president of ProtoTest, a Denver-based testing services company, with both financial success and an excellent track record of client satisfaction. One of the reasons that I’ve agreed to join Pete in this venture is because of his strong commitment to open-source software and continuing to respect and support the Watir community.
One of the first things we’ll be doing is to get a new release of Watir out the door. This has been overdue; me and the other core committers have been very busy over the past six months with other work. The most important thing that we’ll be working on is making Watir work better with Firefox. Specifically, we would like to merge the FireWatir code back into the original Watir code base. I still think that the best way forward on this project is to instrument our test suite so that it can be run against either project. This means some intermediate work on the test suite, as well as putting the Watir project itself under continuous integration using CruiseControl.rb.
]]>Hugh McGowan is presenting his work on Rasta a framework for building keyword-driven tests using Ruby and Excel. This open-source framework fills an important gap for the many testers who are looking for a way to easily put their Watir tests into spreadsheets.
TestCo is providing food and Uplogix is providing beverages. If you are planning to attend, please RSVP to Brent Lavelle so we can have an accurate head count when we order the pizza.
6:15 Pizza
7:00 Rasta, Hugh McGowan
Uplogix, Inc.
7600B N. Capital of Texas Hwy.Suite 220
Austin, TX 78731-1189
This is located on 360 between 2222 and Spicewood Springs Road.
Uplogix is next to the apartment complex with the big American flag so if you are coming from the south you will need to make a U-turn on 360 at this complex and get over to the right lane fast. Traffic on 360 from 5 till 7 will be bad.
We will be planning additional meetings of the Austin Watir Users Group. To recieve future announcements, please subscribe to
watir-austin@yahoogroups.com
We are developing a new product line, mostly following XP, but also influenced by Crystal and Lean. We use test-first development, continuous integration, and automated acceptance testing.
The chief architect needs to define and promote a technical vision for our product line, effectively balancing competing concerns such as customizability and upgradeability. We are looking for someone who knows how to work with an Agile team, developing the architecture incrementally, and work with our business stakeholders, understanding the technical requirements and promoting their technical vision.
The Rails developers we are hiring need to be well-versed in test-driven development and related practices of design patterns and object-oriented design.
These are great opportunities to work with a high-performance team on a challenging project. Contact us if you are interested.
Update: Actually these weren't such great opportunities after all. The project was cancelled and the team was let go.
]]>The most direct treatment of this issue is by Hai Ton, who in his experience report from the Agile 2007 conference says:
What would your Analyst team do when torn between meeting Customer versus Developer demands? When their needs conflict with one another, how do you appease them both? The report outlines the successful strategy one team used to decompose their Stories to serve both the development team needs to demonstrate progress as well as address Customer needs to see value.
His report describes that they used epics, stories and chapters. Epics are described in the form "As a ..., I would like ... so that ...". Customers were asked to sort epics into "must have", "should have", "could have" and "won't have" (MoSCoW). Epics were broken down into multiple stories.
Stories were individually written to be "independent", "negotiable", "valuable to customers", "estimable", "small" and "testable" (INVEST). Stories were described in the "Given ..., When... Then... " format.
Chapters were used when a story was too big for an iteration, and yet breaking it into two stories would no longer clearly demonstrate customer value. Their solution was to create chapters, and then to delay demonstrating the development work to customers until all the chapters of a story were completed.
There is a lot more good detail in this report (A Strategy for Balancing Business Value and Story Size, Proceedings of the Agile 2007 Conference, Hai Ton, ThoughtWorks), which unfortunately is not freely available online.
Here are some other discussions related to this topic that I found interesting.
A related idea is a Minimum Marketable Feature. This term comes from Software by Numbers and describes a feature with clear business value that can then be broken down into stories in development. I haven’t read this book, but have been getting recommendations from so many people that I clearly need to get my hands on a copy.
]]>These are all solid publications with great articles by many other authors as well. All free!
]]>This is the main script that converts ruby code to HTML with syntax highlighting:
require 'rio' require 'rubygems' require 'syntax/convertors/html' code = File.read(ARGV[0]) convertor = Syntax::Convertors::HTML.for_syntax "ruby" @code_html = convertor.convert( code ) puts @code_html fn = "#{File.basename(ARGV[0], File.extname(ARGV[0]))}.html" rio(fn) << @code_html
To get this to work for your blog, you will have to do the following:
> gem install syntax > gem install rio
> code2html.rb your-code.rbThis will create your-code.html, which is your code formatted for your blog.