Here at whitepages.com, our Quality Assurance team is divided into two parts. We have a front end team and a back end team. While, as a group, our responsibility is to ensure that every product we ship is to the highest quality possible, our groups differ greatly in the tools we use for testing and the way we tackle our most difficult challenges. This post is about the front end testing processes we use at WhitePages.com.
The Challenge
When we bring in interview candidates for our team, it can be difficult to explain exactly what we do in our day to day work. Sure, we have exciting projects that ship nearly every release, we have partner sites that require regular maintenance and coordinated releases (sometimes in the wee hours of the morning), and we have an endless backlog of minor bugs that we try to cram into any remaining room in the release cycle, but our job is more than that.
Whitepages.com prides itself on uptime, agility and an excellent user experience. Those three goals affect our day to day work far more than anything we could fit into a given release. We power people search on over 3700 websites ranging from sites that get just a few hits a day to others that bring in millions of hits per day.
The challenge to our team of four front end QA engineers is to cover as much functionality as possible across as many sites as possible every release cycle to ensure we deliver a reliable and consistent experience to our users after every rollout.
The key to achieving maximum coverage is a mix of manual tests to test new functionality and the critical use cases, and automated tests to check for regressions and code changes that affect a large number of our sites. The rest of this post is about some of the techniques/tools we use in our day-to-day testing.
The Solutions
Manual Testing
Over the years, we’ve found the best way to do the basic bug verification for our standard releases is through manual testing. It’s the quickest way to try out the functionality from the user’s perspective and make sure it does what its supposed to do. The exception here is for changes that are in common code that affect all sites, in those cases we’ll check our top tier of sites manually then usually write up some quick tests using Perl or Selenium to test the changes on the rest of the impacted sites. Anyone who has worked on front-end web apps knows that manual testing can be a nightmare though. The matrix of operating system vs. browser vs. resolution, etc can make for a mind-boggling number of test combinations.
Browser usage by our users breaks down something like this:
- Internet Explorer 7 – 45%
- Internet Explorer 6 – 40%
- Firefox 2.0/3.0 – 10%
- Safari – 1.5%
- Other – 3.5%
Given that, we tend to do about 80% of our manual testing in IE. Unfortunately, debugging tools in IE aren’t so great. We tend to rely heavily on the debugging tools/extensions in Firefox such as Firebug, LiveHTTP Headers and Web Developer. Once we’ve debugged the issue and have a fix, we do our final testing and verification using IE.
Each release we also have a suite of manual tests we run to cover critical functionality. While the manual suite is a bit tedious to execute week in and week out, we’ve found the manual high-level pass to be a critical step to wrapping up our testing for the release.
Automated Testing
WhitePages.com is a predominately a Perl shop. The code that powers our site is written in Perl and a lot of the test automation code is written in Perl as well. Everyone on our front-end QA team knows Perl. We all write the tests, we all execute the tests, we all debug the tests. When it comes to writing front end automation tests in Perl, CPAN is our friend:
LWP is a great way to write some quick tests to grab HTML source, scan through it for content and return a pass/fail based on the presence of a given bit of code or text.
WWW::Mechanize makes form handling a breeze and has some excellent built-in functions for checking links and analyzing image elements (check the URLs, check ALT text, etc).
XML::Simple provides a quick way to pull down basic XML and parse through it for a given value without a lot of overhead.
All of the modules above basically request the source of the page and use that to determine whether a given test passes or fails. Often times, that isn’t enough for us. Our users don’t look at the source, our users look at the rendered page. Because of this, we often need to test things in an automated fashion from the user’s perspective.
For automation through the browser, Selenium is what we’ve found works best for us. Its open source, it allows you to test in multiple browsers and, the best part, you can drive it through a Perl script using the WWW::Selenium module on CPAN. We’re working on some very interesting projects using Selenium including a model-based web testing framework to test our rendered UI and a project leveraging Selenium and the User Agent Switcher extension in Firefox to test our Mobile Site against a wide number of mobile user agents.
Even with the best of tools and techniques, there are some great challenges associated with testing a top 50 website that powers thousands of partner sites and pushes new code to production every few weeks. We’re always looking for new and better ways of testing our products and for sharp people who want to join our team.
