Visual Regression Testing with Playwright

What is Playwright?

Playwright is a NodeJS library, created by Microsoft, which offers very similar features to Puppeteer. Both libraries allow you to automate actions within a browser.

With Playwright, you can start or connect to a Chrome, Edge, Safari or Firefox browser and send commands back and forth. The protocol used for these messages is the DevTools protocol (for Chromium browsers) and a custom protocol for Firefox and WebKit browsers.

The main advantage of using Playwright instead of Puppeteer is that Playwright supports multiple browser vendors, where as Puppeteer only supports Chromium based browsers.

How can I use Playwright to do automated testing?

One of the use-cases in using a browser automation framework, is to use it with a testing framework. Popular testing frameworks, such as Jest and Mocha, already implement support for Playwright testing.

These test frameworks allow you to start or connect to a browser and perform the actions defined within your test cases.

An important thing to keep in mind is that Playwright will run in a headless-mode by default. This means that the UI of the browser will not be visible to the user. While running your automated tests, you will not see the browser perform these actions. Headless mode is used to speed up the Playwright execution: it reduces CPU usage because no UI updates are required and improves the speed of total execution.

What is Visual Regression Testing?

Visual Regression Testing is used to prevent unintentional changes to an application's visual state. Let's say you have a website and you know how the page should look like. You instruct your test framework to take a screenshot of your pixel-perfect page, and tell the framework that the page should always look identical to the screenshot you just took.

In a perfect world, your webpage will always look perfect to the end user, just like the screenshot you took. However, sometimes an update to your website, by a developer or someone else in your team, might cause a (visual) bug.

This could be a DOM element missing on your page, or a CSS change that causes rendering issues, or any other bug resulting in an inferior experience to the website visitor.

This is where the power of Visual Regression Testing shines: it will alert you when an unintended change occurs on your website and causes a visual bug.

Visual Testing Libraries

There are quite some libraries available online, open-sourced and other licensed visual comparison libraries, that help you to set up and perform regression testing.

Popular libraries such as BackstopJS for Puppeteer, or Loki provides developers and QA with powerful techniques to quickly, and easily, set up visual regression testing.

In the case of Playwright testing, Playwright Test provides a built-in Visual Comparison feature to take, and visually compare, screenshots with Playwright.

Configure Visual Comparison Testing with Playwright

Playwright contains a package called Playwright Test, which is built as an utility to run automated tests with Playwright.

When you first run a test, Playwright Test will take a screenshot of the page, a baseline screenshot (or golden image), and save it in a folder.

Subsequent tests will continue to take screenshots and match these, pixel-by-pixel, with the baseline screenshot.

Let's say you saved your golden image as golden.png, a simple assertion would be to use this syntax:

expect(await page.screenshot()).toMatchSnapshot('golden.png');

Where page is an object provided by Playwright Test, trying to match the output of page.screenshot() with the contents of golden.png.

When a test fails, meaning the page has changed its contents, you will receive a test error, indicating the screenshot no longer matches the golden image.

Internally, Playwright Test uses the pixelmatch library to compare screenshots for visual differences.

It is possible to pass several options to this library, to fine-tune the comparison of images. The most important option is the threshold option, which indicates the threshold amount of differences allowed between two images.

A full example of a test case using Visual Comparison Testing:

// example.spec.js
const { test, expect } = require('@playwright/test');

test('example test', async ({ page }) => {
  await page.goto('https://testingbot.com');
  expect(await page.screenshot('.hero')).toMatchSnapshot('hero.png');
});

Comparing binary or text formats

Next to visual comparison of images, Playwright Test is also capable of comparing texts and binary outputs.

Let's say for example that you want to make sure your DOM element keeps using the same text content. A simple way to do this is to snapshot the contents of the DOM element and compare it in later runs with the golden snapshot:

// example.spec.js
const { test, expect } = require('@playwright/test');

test('example test', async ({ page }) => {
  await page.goto('https://testingbot.com');
  expect(await page.textContent('.login')).toMatchSnapshot('login.txt');
});

Conclusion

Playwright Test gives you a basic, but powerful way to (visually) compare webpages. Give it a try - more information available on the Playwright Test Documentation page.