Testing Hotlist Update

What I've Been Using Lately

On my last project we used Watir 1.5.4 (only recently publicly released, but we'd been using it since last fall) and Ruby 1.8.5. There are several problems with Ruby 1.8.6 and therefore I will not use it.

We used Rspec on this last project, and I really liked it. We started with Rspec 0.8, and at that time we looked at the Story Runner, but it clearly wasn't ready for production testing. We later upgraded to Rspec 1.1.1. but did not have time to revisit using the Story Runner. The code seems solid now, and I'm eager to try it out on a real project.

We used Eclipse as our IDE, version 3.3 with the Aptana plugin. There is a new version of Aptana that I plan to install (standalone) when I get a moment. Our developers used IntelliJ and really liked it. I was able to get a comp license for it, but haven't actually installed it yet. I think I'm reluctant to use a tool that might be beyond the reach of our users.

Reintegrating Watir and FireWatir

Right now the top priority of the Watir project is to reintegrate with FireWatir and improve the compatability between the two implementations. I've begun serious discussions with Angrez Singh, the FireWatir lead, and we've agreed on this basic objective. There are, however, lots of logistical and architectural issues that need to be discussed further. We've agreed to continue this discussion on the Watir Development list, wtr-development@rubyforge.org.

If you would like to be part of this discussion, or learn more about our plans, please join this list. This is a public list that has been quiet lately, but has been used for similar discussions in the past.

Rasta supports data-driven testing

Rasta is a data-driven testing framework that pulls data out of Excel spreadsheets and executes it using simple "fixtures" that you write in Ruby. Your fixture methods could use Watir or another Ruby-based library (Selenium-RC, Mechanize, Soap4R) to execute against the target application.

There are several things that I think are simply terrific about Rasta. The most dramatic feature is that the spreadsheet is annotated in real-time as the tests execute. Each cell is colored red or green to indicate whether that step passed. If there are errors, they are attached to the cell as a comment that appears when you hover over it.

There are a lot of data-driven testing frameworks that work off of tables these days. Most use wiki's to create the tables, which feels somewhat barbaric. Rasta allows you to use a spreadsheet instead.

Another great thing about Rasta is that it allows test cases to be defined either horizontally or vertically. I've noticed that most data-driven frameworks (Fit, TestFrame, Certify, QTP) specify test cases horizontally. However, most testers, if they are starting from scratch will organize their test matrices vertically. I'm sure there is some psychological reason for this. In any case, Rasta allows you to organize your tests either way. To me this is simply a great of example of how test tools should be adapted to the way tester's think rather than the other way around.

Another feature along these lines is the ability to stop the tests and then pick up where they left off, instead of having to restart from the beginning.

Right now, Rasta only supports Excel, but the developers are working on adding support for Open Office and Google spreadsheet. They've put together a great website, with screen shots and examples. It really gives a much better impression of what the tool is like than I can convey here.

There has been a lot of discussion in the Watir community about data-driven testing frameworks. Any one interested really should give Rasta a look. It is simple, effective, and fun.

Supporting Watir

Pete Dignan and I are founding a company to support Watir and other open-source testing tools. We are still working on the business plan and name of the business. Normally, I would have wanted to wait until we had a few more details decided before making an announcement, but our plans were leaked out early so there is really no point in keeping quiet.

Pete is the president of ProtoTest, a Denver-based testing services company, with both financial success and an excellent track record of client satisfaction. One of the reasons that I’ve agreed to join Pete in this venture is because of his strong commitment to open-source software and continuing to respect and support the Watir community.

One of the first things we’ll be doing is to get a new release of Watir out the door. This has been overdue; me and the other core committers have been very busy over the past six months with other work. The most important thing that we’ll be working on is making Watir work better with Firefox. Specifically, we would like to merge the FireWatir code back into the original Watir code base. I still think that the best way forward on this project is to instrument our test suite so that it can be run against either project. This means some intermediate work on the test suite, as well as putting the Watir project itself under continuous integration using CruiseControl.rb.

Austin Watir Users Group, Weds Mar 19, Keyword Testing with Rasta

I am very please to announce that the first meeting of the Austin Watir Users Group will be this Wednesday, March 19. I've been working for some time with several people to make this happen.

Keyword Driven Testing With Rasta, Hugh McGowan, Convio

Hugh McGowan is presenting his work on Rasta a framework for building keyword-driven tests using Ruby and Excel. This open-source framework fills an important gap for the many testers who are looking for a way to easily put their Watir tests into spreadsheets.

Sponsors

TestCo is providing food and Uplogix is providing beverages. If you are planning to attend, please RSVP to Brent Lavelle so we can have an accurate head count when we order the pizza.

Agenda

6:15 Pizza
7:00 Rasta, Hugh McGowan

Location

Uplogix, Inc.
7600B N. Capital of Texas Hwy.Suite 220
Austin, TX  78731-1189

Driving Directions

This is located on 360 between 2222 and Spicewood Springs Road.

Uplogix is next to the apartment complex with the big American flag so if you are coming from the south you will need to make a U-turn on 360 at this complex and get over to the right lane fast. Traffic on 360 from 5 till 7 will be bad.

Austin Watir Users Group

We will be planning additional meetings of the Austin Watir Users Group. To recieve future announcements, please subscribe to
watir-austin@yahoogroups.com

Looking for Chief Architect and Senior Rails Developers

Due to recent turnover, my team has senior development leadership positions open. These are great opportunities for people who are committed to Agile development and Ruby on Rails.

We are developing a new product line, mostly following XP, but also influenced by Crystal and Lean. We use test-first development, continuous integration, and automated acceptance testing.

The chief architect needs to define and promote a technical vision for our product line, effectively balancing competing concerns such as customizability and upgradeability. We are looking for someone who knows how to work with an Agile team, developing the architecture incrementally, and work with our business stakeholders, understanding the technical requirements and promoting their technical vision.

The Rails developers we are hiring need to be well-versed in test-driven development and related practices of design patterns and object-oriented design.

These are great opportunities to work with a high-performance team on a challenging project. Contact us if you are interested.

Update: Actually these weren't such great opportunities after all. The project was cancelled and the team was let go.

How Big Should a Story Be?

Our recent planning sessions have demonstrated two conflicting forces on story size. The need of customers to see business value and high-level plans pushes towards larger stories, yet the need for developers to be able to complete stories in a single iteration pushes for smaller sizes. I have researched a number of sources to understand more how Agile teams handle this conflict.

The most direct treatment of this issue is by Hai Ton, who in his experience report from the Agile 2007 conference says:

What would your Analyst team do when torn between meeting Customer versus Developer demands? When their needs conflict with one another, how do you appease them both? The report outlines the successful strategy one team used to decompose their Stories to serve both the development team needs to demonstrate progress as well as address Customer needs to see value.

His report describes that they used epics, stories and chapters. Epics are described in the form "As a ..., I would like ... so that ...". Customers were asked to sort epics into "must have", "should have", "could have" and "won't have" (MoSCoW). Epics were broken down into multiple stories.

Stories were individually written to be "independent", "negotiable", "valuable to customers", "estimable", "small" and "testable" (INVEST). Stories were described in the "Given ..., When... Then... " format.

Chapters were used when a story was too big for an iteration, and yet breaking it into two stories would no longer clearly demonstrate customer value. Their solution was to create chapters, and then to delay demonstrating the development work to customers until all the chapters of a story were completed.

There is a lot more good detail in this report (A Strategy for Balancing Business Value and Story Size, Proceedings of the Agile 2007 Conference, Hai Ton, ThoughtWorks), which unfortunately is not freely available online.

Here are some other discussions related to this topic that I found interesting.

• Stories have been getting smaller with time, Jeff Patton
• Consider making multiple passes at a story, to allow for feedback, Alistair Cockburn and Jeff Patton
• Group stories into themes, Dave Larabee
• Timebox the estimating process to allow large backlogs to be estimated (with timeclock), Aslak Hellesoy

 A related idea is a Minimum Marketable Feature. This term comes from Software by Numbers and describes a feature with clear business value that can then be broken down into stories in development. I haven’t read this book, but have been getting recommendations from so many people that I clearly need to get my hands on a copy.

My Writing Process

I've been using the Fieldstone writing method for years. I learned it from James Bach, who learned it from Jerry Weinberg. I've found it to be a great way to generate material, but have struggled over the years to organize the all the writing that it produces. Jerry recently published a book describing the Fieldstone method, but it hasn't helped me much with this problem. What has helped me are a couple of books by David Allen. In Getting Things Done, he describes a process for organizing and processing your email, to-do lists and all the reminders you need to accomplish your goals. I had to read it twice, about six months apart to understand it — it’s simple but deep — and initially used his method simply to organize my to-do lists. Like Weinberg, he encourages you to write whatever is on your mind. For a while, this only lead me to create stacks and stacks of cards. More recently I have learned about other GTD enthusiasts using #7 coin envelopes and floppy-disk binder sheets to organize index cards. This has really worked well for me. I've also recently read Allen's Ready for Anything, which expands on the principles and theory behind the GTD system. This has really helped me adapt it to my writing needs, because writing a book ends up being just another thing to get done, and Allen recommends that you put everything into one system anyway. I’ve also been helped by NoteLens, a simple, free tool for organizing multiple files that Jerry recently recommended.

Cem Kaner's Recent Writing

If you are like me, you’re always eager to read Cem Kaner's writing. Lately, he's been publishing in journals that you may have missed, so here are some links.

• On teaching software testing online, in the December 2007 edition of the journal of the Association for Software Testing.
• On certification and the testing job market, in the June 2007 edition of the journal of the AST.
• On the problems with testers acting as user advocates, in the April 2007 edition of the journal of the Toronto Association of Systems & Software Quality.

These are all solid publications with great articles by many other authors as well. All free!

Publishing Ruby Code

Sometimes I publish Ruby code in this blog. I find that syntax highlighting makes the code much more readable. This post describes the tools I use to make this happen. This approach is based on one by Jim Morris.

This is the main script that converts ruby code to HTML with syntax highlighting:

require 'rio'
require 'rubygems'
require 'syntax/convertors/html'

code = File.read(ARGV[0])

convertor = Syntax::Convertors::HTML.for_syntax "ruby"
@code_html = convertor.convert( code )

puts @code_html

fn = "#{File.basename(ARGV[0], File.extname(ARGV[0]))}.html"
rio(fn) << @code_html

To get this to work for your blog, you will have to do the following:

1. Store the above script as code2html.rb.
2. Install the 'syntax' and 'rio' gems.

   > gem install syntax
   > gem install rio

3. Execute the script as follows:

   > code2html.rb your-code.rb

   This will create your-code.html, which is your code formatted for your blog.
4. Add Jim’s CSS to your blog. 
5. Paste the formatted HTML into your post.

Why Write?

Jerry Weinberg says:

One way for smart people to be happy is to express themselves, to put out in the world the vast melange of thoughts and feelings whirling in their heads.

Next AWTA is postponed

I've talked to many about hosting the AWTA workshop again this January, but I've decided not to go forward with it, not now. I am just too busy. I am, however, talking to a couple of other AWTA regulars — Carl Erickson and Andy Tinkham — about hosting an AWTA workshop in Minneapolis later this spring. The topic will be Testing with Ruby and this will be the first time AWTA has been held outside of Austin. We'll let you know more as we have more details.

I'm letting Carl and Andy take the lead for the next AWTA. That means they get to pick the location, which is why it will be in Minneapolis. Andy lives there, and Carl lives in Grand Rapids, Michigan — which is closer to Minneapolis than it is to Austin. One of the reasons I was happy to have them take the lead is to encourage me to write up the policies and protocols that I have used to organize the workshops in the past. I've been collecting notes for a while. I enjoy small conferences and, like Brian Marick, would like to see more of them. There are a number of different formats that small workshops and conferences can use. AWTA is based on the LAWST model and has evolved over the years to incorporate elements from Open Space as well.

I'll write more about how we’ve organized and hosted the AWTA workshops. If you have specific questions about the format, please post them in comments, and I'll answer them in future posts.

Harry Potter, Test Manager

Jerry Weinberg describes how he’d write a Harry Potter story:

I suspect my Harry Potter would be a test manager, expected to do magic but discounted by software developers because "he's only a tester." As for Voldemort, I think he's any project manager who can't say "no" or hear what Harry is telling him.

The entire interview is quite interesting.

Thoughts on FireWatir

Over the past week, I've taken a close look at FireWatir, with two questions in mind. First, can we make use of it at Dovetail to execute our tests? We have a Watir-based test suite that has over 50 tests and runs in nearly 5 minutes. We've developed it over the past couple of months for a new product. Many of our developers are developing on Macs, so they have a particular interest in using Firefox for testing. Our app needs to support both IE and Firefox.

Secondly, what is the best way to get FireWatir in sync with Watir 1.5 and how should we maintain it going forward? This is a harder question, for a number of reasons that I will try to explain.

FireWatir is based on Watir 1.4. Angrez Singh and the Firewatir team took a copy of the Watir 1.4 code base and then changed it so that it would work with Firefox instead. A lot of the Watir code was reused. It essential amounts to a fork of Watir 1.4. In the past two years since, the Watir code base has evolved significantly. We've cleaned up a lot of the code (both library and test code), refactored much of the code to remove duplication and added support for new attributes and methods (including multiple-attribute support). Few of these changes have been backported to the Firewatir code base. For me, reading through the Firewatir code during the past week has often felt like a trip back in time.

The more that I've reviewed it, the more that I've realized that maintaining FireWatir as a separate code base is a dead end. Some of the classes are direct copies of the original Watir classes (some of which have since changed), some have minor modifications, others are completely changed. Clearly the right way to support this going forward would be to share the classes that can be the same, and refactor the code to maximize the use of shared classes. Indeed, Watir needs a pluggable architecture that will allow adapters for particular browsers, thus supporting SafariWatir and Selenium-RC integration as well. Figuring out how to do this is not trivial. At the same time, it is consistent with the kind of technical refactoring we've been doing for some time in the Watir 1.5 code base. Incremental refactoring of both the code bases is possible because they both have comprehensive test suites that we can use to ensure that the necessary architectural changes don't break the software.

The place to start integrating the two code bases is probably with with tests themselves. Very few changes should be needed to allow the tests to run against either Watir::IE or FireWatir. After reviewing the code, it is pretty clear that running the Watir tests against Firewatir would result in dozens of failures, mostly indications of gaps between where Watir and FireWatir stand today. But synchronizing the test suite and providing a means to mark individual tests as specific to IE or Firefox is really the only way to ensure that the two versions stay in sync.

Some have asked about the core mechanism used in FireWatir. FireWatir requires a Firefox plugin called the Javascript Shell or JSSH. This plugin provides a console interface to the Javascript running in the browser. Once installed, you can actually telnet into Firefox and execute javascript commands. FireWatir connects to this interface using a socket. The basic mechanism is powerful, because it means that anything that can be done in Javascript can be done with FireWatir. However it does require that the plug in be installed first. JSSH is currently provided for Windows (32 bit), Linux and Mac. It is not currently available for FireFox 3.0 (Gran Paradiso) currently under development. Althought it was apparently development independently, it seems that FireWatir users are currently the only users of JSSH. The JSSH code base is open-source, and indeed part of the Firefox code base, but it is unclear to me whether anyone is committed to maintaining this code going forward. This seems to me to be the greatest long-term risk as regards FireWatir. Also the various builds of JSSH that are available appear to have been compiled by different people and different times and they do not seem to have a common or consistent convention regarding versioning. I would feel a lot better if this had some more organizition behind it. Perhaps this could be sponsored by the Mozilla Foundation.

Going forward with developing a single code base supporting two browsers will require us to have a stricter testing policy that we've had so far. Installing CruiseControl.rb on a Windows virtual server will allow us to automatically run our tests on both IE and Firefox whenever code changes are made. The results would be public, so any one could see them. Having this in place makes it easier for us to coordinate the work of multiple contributors who may be using different development platforms. A hosted Windows virtual server, which is what we would need to run our tests, costs about $50 a month. This makes me think that it may now be time to pass the hat and ask users to help fund Watir development going forward. I'm not sure if I should simply ask people to send me money (say, using a paypal button), or whether we should go through the formality of setting up a non-profit or selling memberships in a society. I'd appreciate your thoughts on this. Jason Darling suggests T-shirts sales.

In working with FireWatir this week, I spent a lot of time struggling with a speed issue. My tests would run very slowly, especially when typing. They seemed slower than what a manual tester would do. Tests would run fine for a bit, and then would suddenly slow down, making me think that some kind of resource had been exhausted. Although it showed up slightly differently in different circumstances, I reproduced the problem on three different machines, with both XP and Vista, and both with the Dovetail test suite and the FireWatir unit tests. I have also spoken with both Paul Rogers and Jim Matthews, both of who have made significant use of FireWatir and have not seen this problem. Indeed, Jim says that FireWatir is faster than Watir for him. I eventually found a workaround for my speed problem, but it remains an issue of concern for me. I don't know whether the problem lies in FireWatir itself or in the JSSH plugin. The next step would be to add a debugging/logging layer to FireWatir to help debug these and other issues.

In running the Dovetail tests, I ran into five specific errors, blocking about 90% of the tests. Three were obviously caused by missing features new to Watir 1.5. I suspect the other two were also in this category, but did not have time to track them down far enough. In investigating these issues, I noticed that the attach command was missing. I use this all the time to debug tests. Without it, my ability to investigate the source of errors was limited. This feature was included in Watir 1.4; I'm not sure why it isn't in FireWatir. The missing features I looked into could be added to FireWatir trivially. Merging the code bases would accomplish this as a side effect.

Looking forward, I think it will be a long term project to integrate Watir and FireWatir. We don't have many contributors, all of us mostly work on Watir on our own time. It took us two years to go from 1.4 to 1.5 and I think it will be a similar timeframe to see the complete integration of Watir and FireWatir.