How to programmatically test visual design of components

The Problem

When writing component based applications or libraries we often create components where part of their API only changes the appearance.

<Button primary>toggle primary prop</Button>

<Button>toggle primary prop</Button>

toggle primary prop

How should we best test this?
Lets find out!

First Identify the interface(s)

Before writing any test we should be clear about the interfaces of an abstraction and their consumers.
This is a big topic for another post... jumping to conclusions:

The abstraction is the Button component.
It's interfaces are:
1. The programming Interface.
  (In react: props + context + return value)
2. It's UI + UX
Their consumers are:
1. Fellow programmers using the component in their app
2. The users of this app

Most abstractions in frontend applications have two kinds of consumers: Engineers & App-Users

Then find the cause + effect you want to test

For our Button:

When the primary prop is set to true by an engineer [via complex hooks and stuff] (cause)
Then the background of the app should appear fuchsia to the app-user (effect)

fuchsia is the main brand color!

Now decide on the best tooling to write an optimal programmatic test with

This is where things get tricky! Most tools I see being used today don't support writing optimal tests. So let's compare them!

ℹ️ Characteristics of an optimal test

The test acts on the cause and asserts on the effect (see AAA & previous section)
The test does not cover implementation details (see: Testing Implementation Details)
It only breaks when the covered effect changes.

So what are the options?

Option 0: Just don't test

I think it's fully valid to just not test visual design programmatically. Manual QA will most likely be enough.

Still there are cases where automated tests are really valuable. Most primarily in design systems/component libraries where the mission is to provide abstractions that are easy to use for engineers and look as intended to the user.

As a rule of thumb ask yourself: "Am I (or my team) the only one using this component?"

Yes: Only Add tests if you feel the component is unstable.
No: Add tests unless the component is dead simple.

Option 1: Snapshot Testing

A simplified snapshot test would look like this:

expect(renderToHTML(<Button primary>Hello</Button>))
  .toMatchInlineSnapshot(`
    "<button class="primary">Hello<button>"
  `);

From my experience tests like this have little to no value for appearance testing because they assert on an implementation detail (namely the class name).

👎 the primary class could set background: olive; but the test would be green.
👎 refactoring to <button style="background: fuchsia;"> would have the same effect but the test would fail.
👎 refactoring the class name or adding others would make the test fail.

This does not mean snapshot tests are bad. I recommend reading Effective Snapshot Testing.

Option 2: Visual Regression Testing

A simplified visual regression test would create a screenshot of the button rendered in a browser and compare it to a stored screenshot. If any pixels have changed, the test would fail.

Illustration of a document diff Image Source

While this perfectly covers the effect without depending on implementation details, it fails on focussing on the single effect we want to test. Thereby violating point 3 of an optimal test.

👎 Any change to the looks of the button will now cause the test to fail. All buttons now have a border? All button tests fail!
👎 Even changing the button label will fail the test.
👎 Visual regression tests also fully cover the browsers rendering/painting process.

Assertions have the highest value when they cover effects as short as possible after they leave your abstraction. If we let the effects flow though a bunch of other systems, chances are they get blurry.

For example we will not test that the primary background color is actually perceived as "fuchsia" by the user. (That should have been done beforehand by user- and a11y-tests).

Why should we test that the browser actually renders a DOM-node with a backgroundColor of fuchsia accordingly? That is something that the maintainers of the browser should test, not us.

This also does not mean visual regression tests are bad in general. I recommend reading Visual Regression Testing.

Option 3: Style testing

The closes point where the effect leaves our domain (the point after which we can not accidentally introduce bugs anymore) is the computed styles of the DOM-Node in the browser.

Let's test that!

import { getRealStyles, toCss } from 'test-real-styles';

describe('primary prop', () => {
  it('makes the background fuchsia', async () => {
    const button = renderToDomNode(<Button primary>Hello</Button>);

    const styles = await getRealStyles({
      css: readFileSync('button.css'), 
      doc: button,
      getStyles: ['backgroundColor'],
    });
    expect(toCss(styles)).toMatchInlineSnapshot(`
      "background-color: fuchsia;"
    `);
  });
});

👍 Acts on the cause (primary prop) and asserts on the effect (background-color)
👍 Doesn't cover implementation details on how the style is applied to the element
👎 Still covers implementation details when it comes to element composition. (refactoring the button to be wrapped in a span will fail the test)
👍 Adding other styles or changing the label will not break the test.

I shamelessly recommend using test-real-styles because it uses real browsers and supports using your full page css.

There is also toHaveStyle of jest-dom but it did not work for me because it keeps the element in jsdom which has lots of issues emulating CSSOM

Most css-in-js solutions provide something like jest-styled-components which is pretty neat, too. (A downside is that these will also not evaluate the styles in a real browser and therefore will not capture bugs caused by the cascade.)

Finally write the actual test

Given we decided on a style-testing solution, I want to share a few thoughts that might make the tests even more valuable.

Writing .button { background: fuchsia; } in the implementation and expect(getStyle('background')) .toBe('fuchsia') in the test feels tedious.

Though in bigger components this will be more meaningful and stabilizing, there is a potential gem to be created:

A machine-readable spec of low level design patterns

Sit down with your designers and not only talk about how the button should look like but also which underlying patterns lead to this look.

This might surface things like:

"We want all intractable elements to have a border-radius of 5px"
"The main solid interaction element should use our brand-color as background"

Which can be translated to:

// designIntentions.js
import { brandColor } from './atoms';

export const interactable = {
  borderRadius: '5px';
};
export const primaryInteractable = {
  ...interactable,
  backgroundColor: brandColor
}

Which then can be used in style-tests like:

import { getRealStyles } from 'test-real-styles';
import { primaryInteractable } from 'designIntentions.js';

describe('primary prop', () => {
  it('applies primary interactable styles', async () => {
    const buttonElement = renderToHtmlElement(<Button primary>Hello</Button>);

    const styles = await getRealStyles({
      css: readFileSync('button.css'), 
      doc: buttonElement,
      getStyles: Object.keys(primaryIntractable),
    });
    expect(styles).toEqual(primaryIntractable);
  });
});

And I think that's beautiful.

Do you think your project could benefit from style testing?

I'm available to help you set everything up technically, help dev and design teams to get the most value of this and also write tests for you.

get in touch ❣️

The Problem

When writing component based applications or libraries we often create components where part of their API only changes the appearance.

<Button primary>toggle primary prop</Button>

<Button>toggle primary prop</Button>

toggle primary prop

How should we best test this?
Lets find out!

First Identify the interface(s)

Before writing any test we should be clear about the interfaces of an abstraction and their consumers.
This is a big topic for another post... jumping to conclusions:

The abstraction is the Button component.
It's interfaces are:
1. The programming Interface.
  (In react: props + context + return value)
2. It's UI + UX
Their consumers are:
1. Fellow programmers using the component in their app
2. The users of this app

Most abstractions in frontend applications have two kinds of consumers: Engineers & App-Users

Then find the cause + effect you want to test

For our Button:

When the primary prop is set to true by an engineer [via complex hooks and stuff] (cause)
Then the background of the app should appear fuchsia to the app-user (effect)

fuchsia is the main brand color!

Now decide on the best tooling to write an optimal programmatic test with

This is where things get tricky! Most tools I see being used today don't support writing optimal tests. So let's compare them!

ℹ️ Characteristics of an optimal test

The test acts on the cause and asserts on the effect (see AAA & previous section)
The test does not cover implementation details (see: Testing Implementation Details)
It only breaks when the covered effect changes.

So what are the options?

Option 0: Just don't test

I think it's fully valid to just not test visual design programmatically. Manual QA will most likely be enough.

As a rule of thumb ask yourself: "Am I (or my team) the only one using this component?"

Yes: Only Add tests if you feel the component is unstable.
No: Add tests unless the component is dead simple.

Option 1: Snapshot Testing

A simplified snapshot test would look like this:

expect(renderToHTML(<Button primary>Hello</Button>))
  .toMatchInlineSnapshot(`
    "<button class="primary">Hello<button>"
  `);

From my experience tests like this have little to no value for appearance testing because they assert on an implementation detail (namely the class name).

👎 the primary class could set background: olive; but the test would be green.
👎 refactoring to <button style="background: fuchsia;"> would have the same effect but the test would fail.
👎 refactoring the class name or adding others would make the test fail.

This does not mean snapshot tests are bad. I recommend reading Effective Snapshot Testing.

Option 2: Visual Regression Testing

A simplified visual regression test would create a screenshot of the button rendered in a browser and compare it to a stored screenshot. If any pixels have changed, the test would fail.

Illustration of a document diff Image Source

While this perfectly covers the effect without depending on implementation details, it fails on focussing on the single effect we want to test. Thereby violating point 3 of an optimal test.

👎 Any change to the looks of the button will now cause the test to fail. All buttons now have a border? All button tests fail!
👎 Even changing the button label will fail the test.
👎 Visual regression tests also fully cover the browsers rendering/painting process.

Assertions have the highest value when they cover effects as short as possible after they leave your abstraction. If we let the effects flow though a bunch of other systems, chances are they get blurry.

For example we will not test that the primary background color is actually perceived as "fuchsia" by the user. (That should have been done beforehand by user- and a11y-tests).

Why should we test that the browser actually renders a DOM-node with a backgroundColor of fuchsia accordingly? That is something that the maintainers of the browser should test, not us.

This also does not mean visual regression tests are bad in general. I recommend reading Visual Regression Testing.

Option 3: Style testing

The closes point where the effect leaves our domain (the point after which we can not accidentally introduce bugs anymore) is the computed styles of the DOM-Node in the browser.

Let's test that!

import { getRealStyles, toCss } from 'test-real-styles';

describe('primary prop', () => {
  it('makes the background fuchsia', async () => {
    const button = renderToDomNode(<Button primary>Hello</Button>);

    const styles = await getRealStyles({
      css: readFileSync('button.css'), 
      doc: button,
      getStyles: ['backgroundColor'],
    });
    expect(toCss(styles)).toMatchInlineSnapshot(`
      "background-color: fuchsia;"
    `);
  });
});

👍 Acts on the cause (primary prop) and asserts on the effect (background-color)
👍 Doesn't cover implementation details on how the style is applied to the element
👎 Still covers implementation details when it comes to element composition. (refactoring the button to be wrapped in a span will fail the test)
👍 Adding other styles or changing the label will not break the test.

I shamelessly recommend using test-real-styles because it uses real browsers and supports using your full page css.

There is also toHaveStyle of jest-dom but it did not work for me because it keeps the element in jsdom which has lots of issues emulating CSSOM

Finally write the actual test

Given we decided on a style-testing solution, I want to share a few thoughts that might make the tests even more valuable.

Writing .button { background: fuchsia; } in the implementation and expect(getStyle('background')) .toBe('fuchsia') in the test feels tedious.

Though in bigger components this will be more meaningful and stabilizing, there is a potential gem to be created:

A machine-readable spec of low level design patterns

Sit down with your designers and not only talk about how the button should look like but also which underlying patterns lead to this look.

This might surface things like:

"We want all intractable elements to have a border-radius of 5px"
"The main solid interaction element should use our brand-color as background"

Which can be translated to:

// designIntentions.js
import { brandColor } from './atoms';

export const interactable = {
  borderRadius: '5px';
};
export const primaryInteractable = {
  ...interactable,
  backgroundColor: brandColor
}

Which then can be used in style-tests like:

import { getRealStyles } from 'test-real-styles';
import { primaryInteractable } from 'designIntentions.js';

describe('primary prop', () => {
  it('applies primary interactable styles', async () => {
    const buttonElement = renderToHtmlElement(<Button primary>Hello</Button>);

    const styles = await getRealStyles({
      css: readFileSync('button.css'), 
      doc: buttonElement,
      getStyles: Object.keys(primaryIntractable),
    });
    expect(styles).toEqual(primaryIntractable);
  });
});

And I think that's beautiful.

Do you think your project could benefit from style testing?

I'm available to help you set everything up technically, help dev and design teams to get the most value of this and also write tests for you.

get in touch ❣️

The Problem

First Identify the interface(s)

Then find the cause + effect you want to test

Now decide on the best tooling to write an optimal programmatic test with

ℹ️ Characteristics of an optimal test

Option 0: Just don't test

Option 1: Snapshot Testing

Option 2: Visual Regression Testing

Option 3: Style testing

Finally write the actual test

A machine-readable spec of low level design patterns

Do you think your project could benefit from style testing?

▇▇▇▇▇▇▇▇▇▇▇

How to programmatically test visual design of components

The Problem

First Identify the interface(s)

Then find the cause + effect you want to test

Now decide on the best tooling to write an optimal programmatic test with

ℹ️ Characteristics of an optimal test

Option 0: Just don't test

Option 1: Snapshot Testing

Option 2: Visual Regression Testing

Option 3: Style testing

Finally write the actual test

A machine-readable spec of low level design patterns

Do you think your project could benefit from style testing?