Automating CSS Regression Testing
The following is a guest post by Garris Shipon . We've touched on the four types of CSS testing here before. Regression testing is the hardest. It's the type where you're trying to test if a change you made to CSS resulted in any unexpected visual problems. This is made more difficult with responsive designs. Garris built a tool for doing this as he embarked upon a new responsive design for a large scale site. Here, he'll walk you through the whole thing.
Why CSS testing?
Do a search for CSS regression testing and a common theme becomes clear – breaking CSS is easy, testing it is hard. This was the case at the onset of a large responsive refactoring project I scoped earlier this year for Art.com.
Like many other established web companies, Art.com is in the process of adding responsive behavior to its massive e-commerce web app, which was originally designed for 1024px desktop screens.
I realized this would be a big and delicate job. Adding multiple breakpoint behaviors which (by their very nature) are mostly off-screen at any point in time means we would have a lot of hard-to-find errors. We needed a way for our engineers to automate bug discovery before slamming our QA process with hundreds of ticky-tacky little layout issues (which would all need to be documented, tracked and fixed).
CSS testing options
Looking at existing options for CSS regression coverage, I discovered that most came with a lot of integration overhead. To me, these solutions seem more geared for QA engineers who are focused exclusively on testing. These testing apps have deep and wide coverage potential but are too complex and bulky for most UI engineers to easily integrate into their development process.
The solution I wanted would be easy to install locally, use familiar syntax and give us a reasonable amount of confidence that a selector change I make for mobile isn't going to result in a hard-to-find bug in my desktop layout.
Of the existing solutions, the one which had the most potential for my needs was phantomCSSwritten by James Cryer . This is an awesome tool for visually testing complex UI module interaction states, but it requires full-on CasperJS scripting which is still unnecessarily cumbersome for testing large sets of static CMS templates at different screen dimensions.
This situation was the impetus for creating BackstopJS , which is essentially a wrapper forResembleJS (also written by James Cryer ) that enables an easy-to-configure visual regression test matrix across multiple URLs, page elements and screen sizes.
The following is a 15 minute walk-through of an installation and initial configuration of BackstopJS.
Installation
BackstopJS can be downloaded as a .zip file from GitHub, but it's best to install it using the super useful Bower package manager . The rest of the instructional will be based on the Bower install and a simple demo project (download zip), which is taken directly directly the Bootstrap example page .
Node and the NPM package manager are foundational dependencies also required for the rest of the install. It's possible you already have these installed (from other projects). If not, I'd really recommend you to learn more . There are really amazing developments going on in this ecosystem!
Expand the simple demo
Lets start by downloading the demo project files . Or, you can use your own active project. We will install the testing framework right into here:
Install BackstopJS with Bower & NPM
Go to your project root, in this case: myCoolProject
$ cd ~/path-to-myProjects/myCoolProject
$ bower install backstopjsYour directory should look like this:
Install dependencies
So far so good, you now have BackstopJS scaffolding, but there are some super useful packages that BackstopJS depends on so we need to install those.
Install local dependencies with NPM
Node and the NPM package manager and required for the rest of the install.
$ cd bower_components/backstopjs
$ npm installIf you don't already have a global Gulp instance...
Gulp is a great system for enabling various task automation. It's great for project building, asset minification, CSS preprocessing etc. It is a more functional way of tackling the kinds of tasks Grunt became popular for.
$ sudo npm install gulp -gIf you don't already have a global PhantomJS install...
Well then, this is your lucky day! PhantomJS is a headless web browser. It does everything a web browser does, except display content to the screen.
$ sudo npm install phantomjsIf you don't already have a global CasperJS install...
Then don't miss this. CasperJS ( installation docs ) does for headless web clients what jQuery did for DOM selection.
$ sudo npm install -g casperjs
Install complete!
Still with me? If you have made it this far then you have (among other things) some badass CSS regression testing ingredients. Lets see what we can do.
Generate a test configuration template
From the myCoolProject/bower_components/backstopjs directory:
$ gulp genConfigThis will generate a boilerplate JSON configuration file for you at the root of your project e.g. `myCoolProject/backstop.json`. This is where you can specify the URL's, selectors to test and at which breakpoints you want to test them at.
{
"viewports" : [
{
"name": "phone",
"viewport": {"width": 320, "height": 480}
}
,{
"name": "tablet_v",
"viewport": {"width": 568, "height": 1024}
}
,{
"name": "tablet_h",
"viewport": {"width": 1024, "height": 768}
}
]
,"grabConfigs" : [
{
"testName":"http://getbootstrap.com"
,"url":"http://getbootstrap.com"
,"hideSelectors": [
]
,"removeSelectors": [
"#carbonads-container"
]
,"selectors":[
"header"
,"main"
,"body .bs-docs-featurette:nth-of-type(1)"
,"body .bs-docs-featurette:nth-of-type(2)"
,"footer"
,"body"
]
}
]
}In this configuration you can see three viewports objects, one for phone , tablet vertical andtablet horizontal , each with a name and a viewport property. You can add as manyviewports objects as you need. BackstopJS requires at least one .
Then we have grabConfigs . This specifies the URLs and element selectors that BackstopJS will test. The selectors generally accept standard CSS selector notation. It's useful to think of everygrabConfigs object as a test for a specific static page or global app state. Here too, you can add as many grabConfigs objects as you need. BackstopJS requires at least one .
You may notice that in this config, our URL is pointing to http://getbootstrap.com (the homepage of the Bootstrap design component library) and that is what we would see if we were to run our test now. This is here to illustrate that BackstopJS can point to local or remote URLs so it's easy to imagine repurposing the same tests for local development, QA, staging and production environments.
Modify the configuration template
For our demo we want to test our local project file, so please make the following change replacing the grabConfigs node in `myCoolProject/backstop.json`.
,"grabConfigs" : [
{
"testName":"http://getbootstrap.com"
,"url":"../../index.html"
,"hideSelectors": [
]
,"removeSelectors": [
]
,"selectors":[
"nav"
,".jumbotron"
,"body .col-md-4:nth-of-type(1)"
,"body .col-md-4:nth-of-type(2)"
,"body .col-md-4:nth-of-type(3)"
,"footer"
]
}
]generating (or updating) reference screenshots
From the `myCoolProject/bower_components/backstopjs` directory:
$ gulp referenceThis task will create a (or update an existing) bitmaps_reference directory with screen captures representing all specified selectors at every breakpoint. Take a look inside `bower_components/backstopjs/bitmaps_reference`:
So far so good. We now have our reference set. Now lets run a test!
Running our first test
We are about to run our first test. But keep in mind, we haven't changed anything in our project yet. So our tests should pass.
From the `myCoolProject/bower_components/backstopjs` directory:
$ gulp testThis task will create a new set of images in `myCoolProject/bower_components/backstopjs/bitmaps_test/<timestamp>/`.
Once the test images are generated, a web browser should open and a report comparing the most recent test bitmaps against the current reference images will run. Significant differences will be detected and shown.
Test screen shots should have been generated and a clean test report should be displayed. Now let's try changing our CSS and see what happens.
Updating our index file and running our second test
Open our test file at `myCoolProject/index.html` and add the following code somewhere in the<head> tag of the markup:
<style>
.jumbotron {
padding: 0px;
}
</style>Now, From the `myCoolProject/bower_components/backstopjs` directory.
$ gulp testOur test should run again and errors should be found, scroll the report down to see a visual diff of the problems.
Our visual diff contains the reference capture, the most recent test capture and the visual diff file.
And there you have it – regression found!
This is a very simple example. In real life, designers and engineers may find themselves working on very complex responsive layouts. That is when a system like this really provides value. By automating the repetitive tasks we can move our CSS projects forward by focusing on more complex and creative tasks that can't be automated.
There are a lot of things I can imagine for BackstopJS to satisfy more use cases, here are a few obvious ones I would love to have for myself:
- A UI for managing multiple config profiles and associated reference data.
- Firefox and IE rendering
- Currently, screenshots are executed when the native document ready event is fired, it would be more useful if we could specify a custom ready event for things like web apps that may be making Ajax calls or running some other async process before the display can be fully rendered.
- A much easier install process
