Context

Currently CDAP UI has some unit tests, but they are not actively run or maintained. This is because we did not add Bamboo integration when we added these tests, which reduced these tests do not integrate with Bamboo yet, which reduce their usefulness since UI developers had have to manually run the tests before creating or merging a PR. After a while we stopped maintaining these tests as we made changes to existing components, and also stopped adding new tests. Right now UI developers have to do manual testing Also most of these tests are unit tests, which do not cover critical user journeys. Right now to make sure that existing features do not break when adding new features, to make sure that existing features do not break UI developers have to test manually. This has a strong negative impact on developer productivity and velocity. Adding unit/integration/end-to-end UI tests will allow us to have more increase confidence when adding new features, and will also increase developer productivity.

General Approach

In general, there are three kinds of tests, in increasing order of complexity and time each takes to run: unit tests, integrations tests, and end-to-end (e2e)/UI tests. Our testing philosophy is that we want to test features the same way the the user would use them, instead of testing for implementation details. This usually means writing more integration tests, instead of just writing unit tests for each component, since for most of our features we need to dispatch an asynchronous call to fetch or write data.

This does not mean we should not write unit tests. On the contrary, we should have a snapshot test for every React component we write, as this is a quick and easy way to check that the rendered DOM does not change unexpectedly.. All the helper and pure functions we use should also be tested as well, and this includes Redux reducers.

UI developers should run the test suite before creating a PR, and we will also integrate with Bamboo so that tests are kicked off for every new commit to a PR.

Tools

Unit/snapshot testing

Jest

Jest is a Javascript testing framework, that can act as a test runner, assertion library, and mocking library. Some of the best features of Jest include:

• Instant feedback: Fast interactive watch mode runs only test files related to changed files

• Snapshot Testing: Capture snapshots of React trees or other serializable values to simplify testing and analyze how state changes over time.

• Fast and sandboxed: It runs parallel tests across workers, and buffers console messages and prints them together.

• Built-in code coverage reports: Supports —-coverage for bringing out of the box coverage reports.

Jest can also be used to mock functions, which is useful for testing event handlers for example, to see whether the function is called when the event is triggered.

Integration testing

react-testing-library

“Simple and complete React DOM testing utilities that encourage good testing practices.”

This library allows interacting directly with DOM nodes, instead of dealing with instances of rendered React components. The philosophy behind this library is to encourage testing features similar to how users would use them. More specifically, it provides utility functions to find elements by their text, trigger actions, and wait for an element to show up after an async call. This means writing more integration tests, but it will give us more confidence when shipping new features.

One con of this library is that it is from a single developer, but it’s starting to see more and more adoption in the industry. It is also a recommended testing tool, along with Jest and Enzyme, in the official React docs.

Also considered:

Enzyme

Enzyme is a JavaScript testing utility for React that makes it easier to assert, manipulate, and traverse your React components' output. Enzyme’s API for DOM traversal and manipulation is similar to jQuery’s.

While Enzyme is very powerful, its versatility can be a drawback, as it could encourage bad practices. Enzyme allows checking for state and props of a component, however this is testing implementation details, and these are not the things the user can see.

E2E testing

Cypress

Cypress is an all-in-one testing framework focused on e2e tests. It injects itself on to the web app as Javascript scripts, so we can actually see the tests running in the browser as content is rendered. Some of the best features include:

• Takes snapshots as tests run

• Automatically reloads whenever changes are made to tests

• View screenshots taken automatically on failure, or videos of the entire test suite when run headlessly.

• Can debug directly from Chrome DevTools, along with readable errors and stack traces

• Clean API to interact with browser (similar to jQuery’s)

• Is super fast, as fast as the browser can render

Also considered:

Puppeteer

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium. Features include:

• Generate screenshots and PDFs of pages.

• Crawl a SPA and generate pre-rendered content (i.e. "SSR").

• Automate form submission, UI testing, keyboard input, etc.

• Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.

• Capture a timeline trace of your site to help diagnose performance issues.

• Test Chrome Extensions.

One drawback of Puppeteer is that it will only work with Chrome. Cypress also currently only works with Chrome, but the team is actively working to add support for other browsers such as Firefox, Edge, and Safari. Also, the API to interact with the browser is not as clean as Cypress’.

Detailed Approach

Snapshot testing with Jest

Jest provides the ability to do snapshot testing, a very useful tool that helps to make sure UI does not change unexpectedly. The first time a test is run, the output passed into the test is saved to a “snapshot file”. Then when the test is run in the future, the output is compared to the snapshot file. If the output matches the file, the test passes, if the output is different then the file, the test fails and Jest prints a diff.  If we changed internal implementation of the component, we can just press a button (‘u’) to update the snapshot and the test will start passing again.

Snapshots will be saved beside the test file that created them in an auto-generated __snapshots__ folder. We just have to make sure the initial snapshot generated reflects the intended outcome.

Using react-testing-library’s render function and the ‘Page500’ component we have in our library, a snapshot can be created as simply as this:

// components/500/__tests__/500.test.js

import { render } from 'react-testing-library';
import Page500 from 'components/500';

describe('Page500', () => {
  it('renders correctly', () => {
    const { container } = render(<Page500 />);
    expect(container.firstChild).toMatchSnapshot();
  });
});

For the above example, a snapshot will be generated and will look like this:

// components/500/__tests__/__snapshots__/500.test.js.snap

exports[`Page500 renders correctly 1`] = `
<div

  class="page-500"

>

  <h1

    class="error-main-title"

  >

    features.Page500.mainTitle

  </h1>
  ...
</div>

`

This is the rendered output of the fully mounted Page500 component. The h1 text is ‘features.Page500.mainTitle’ instead of the actual text, since in the test we don’t have i18n-react to actually get the text.

Snapshot testing is powerful and simple to set up, so we should have a snapshot test for every single React component. However as mentioned before, we need to verify that each functionality is working as intended when the user uses it, so that’s where react-testing-library comes in.

Integration testing with react-testing-library

React-testing-library provides simple and intuitive APIs to test a component just like how an user would use it. For example, the library provides `fireEvent` action to fire events, `getByText` to find an element by its text (just as how the user would), and `waitForElement` to wait for an element to appear after an async call. React-testing-library also provides a way to tag elements that don’t have text, through `getByTestId`, which will get elements that have `data-testid` attribute set to a specific value/test id.

Here’s one example test for the Tags component, with inline comments.

import React from 'react';
import { cleanup, render, fireEvent, wait, waitForElement } from 'react-testing-library';
import Tags from 'components/Tags';
import 'jest-dom/extend-expect'; // add custom jest matchers from jest-dom

afterEach(cleanup); // unmount and cleanup DOM after test is finished

jest.mock('api/metadata'); // manually mock api/metadata

const testEntity = {
  type: 'application',
  id: 'test'
};

describe('Tags', () => {
  it('adds new tag when user inputs a tag and presses Enter', async () => {
    const TEST_TAG = 'UI_TEST_TAG';
    const { getByTestId, queryByTestId, getByText, container } = render(<Tags entity={testEntity} />);
    await wait(() => !queryByTestId('loading-icon')); // wait until loading icon is gone, which means async call is done

    const plusButtonElem = getByTestId('tag-plus-button');
    fireEvent.click(plusButtonElem, {container}); // click on plus button to show input box
    await waitForElement(() => getByTestId('tag-input'));

    const inputElem = getByTestId('tag-input');
    expect(inputElem).toHaveTextContent('');
    fireEvent.change(inputElem, {target: {value: TEST_TAG}});
    fireEvent.keyPress(inputElem, {key: 'Enter'}); // enter value, and then press Enter to add tag
    await waitForElement(() => getByText(TEST_TAG));

    expect(inputElem).not.toBeInTheDocument(); // input elem is hidden after user presses Enter
    expect(getByText(TEST_TAG)).toBeVisible();
  });
});

Some notes about this test:

1. First we extend Jest matchers through importing `js-dom`, and add an `afterEach` to cleanup the DOM after the test is done, since react-testing-library actually renders the component into the DOM. Both of these are added manually here, but they can be added to a global `jest.setup.js` file which is called before every test.

2. Since in the `Tags` component we reference `MyMetadataApi` to get and add tags, in this test we need to mock this module so that we don’t actually make calls to backend. Since the module is located at `api/metadata`, if we add the mock implementation to `api/__mocks__/metadata` then Jest will automatically pick it up.

3. In the actual test, first we wait until the loading icon (tagged by `data-testid=”loading-icon”`) is null, which means the initial async calls to fetch tags are done. Then we find the plus button and click on it to show the input box. We then type in a tag value, and press Enter. We can then wait and verify that a new tag with the inputted value appears, and the input box is no longer shown in the document.

Testing Redux

Testing Redux-connected components

Many of our React components are connected to the appropriate Redux stores. The connected components are usually the default exports, since they are the ones imported by other components.

These connected components are harder to test, since we need to check that the component is rendered correctly based on the state in its store. Instead of trying to test both the component and the store, we can test the unconnected component instead. We manually pass in the props to the component like in the example code above, and see if the components renders correctly based on those props. We can then test the Redux pieces (actions creators, reducers etc.) individually like below.

Testing Redux reducers

Since most reducers are pure, i.e. they return a state object after taking an action and previous state, they’re quite easy to test. For an example in our code, most stores handle a ‘SET_LOADING’ action. So given an example store like this:

export const Actions = {

  SET_LOADING: ‘SET_LOADING’

};

const initialState = {

  loading: false

};

const testReducer = (state = initialState, action) => {
  switch (action.type) {
    case Actions.SET_LOADING:
      return {

        ...state,

        loading: action.payload.loading,

      }
    default:
      return state
  }
}

… export default TestStore;

We can test the reducer like this:

import TestStore, {Actions} from '../TestStore';
import {defaultAction} from 'services/helpers';

describe('testReducer', () => {
  it('should return the initial state', () => {
    expect(reducer(undefined, defaultAction)).toEqual({loading: false})
  })

  it('should handle SET_LOADING', () => {
    expect(
      reducer({loading: false}, {
        type: Actions.SET_LOADING,
        payload: {loading: true}
      })
    ).toEqual({loading: true})

    expect(
      reducer({loading: true}, {
        type: Actions.SET_LOADING,
        payload: {loading: false}
      })
    ).toEqual({loading: false})
})

Testing Redux action creators

In a lot of places in our code, we have action creators, with actions similar to this:

import TestStore, {Actions} from '../TestStore';

const setStoreLoading = (loading) => {
  TestStore.dispatch({
    type: Actions.SET_LOADING,

    payload: {loading}
  });
};

To test this function, we can use Jest’s spyOn method to test that the dispatch function was called. For example:

import TestStore, {Actions} from '../TestStore';

describe('testSetStoreLoading', () => {
  it('should dispatch the action', () => {
    const spy = jest.spyOn(TestStore, dispatch);
    TestStore.setStoreLoading(true);
    expect(spy).toHaveBeenCalled();
    expect(spy).toHaveBeenCalledWith(true);
  });
});

E2E testing

As mentioned above, Cypress makes it easy to write tests for end-to-end use cases with its simple API. Here’s an example of creating a pipeline with specific configuration (i.e. having Instrumentation value as ‘Off’ instead of ‘On’ like default), then checking that the deployed pipeline has this configuration.

const TEST_PIPELINE_NAME = '__UI_test_pipeline';
const TEST_PATH = '__UI_test_path';

describe('Creating a pipeline', function() {
 it('is configured correctly', function() {
   // Go to Pipelines studio
   cy.visit('/');
   cy.get('#resource-center-btn')
     .click();
   cy.contains('Create')
     .click();
   cy.url()
     .should('include', '/pipelines');

   // Add an action node, to create minimal working pipeline
   cy.get('.item')
     .contains('Conditions and Actions')
     .click();
   cy.get('.item-body-wrapper')
     .contains('File Delete')
     .click();

   // Fill out required Path input field with some test value
   cy.get('.node-configure-btn')
     .invoke('show')
     .click();
   cy.get('.form-group')
     .contains('Path')
     .parent()
     .find('input')
     .click()
     .type(TEST_PATH);

   cy.get('[data-testid=close-config-popover]')
     .click();

   // Click on Configure, toggle Instrumentation value, then Save
   cy.contains('Configure')
     .click();
   cy.get('.label-with-toggle')
     .contains('Instrumentation')
     .parent()
     .as('instrumentationDiv');
   cy.get('@instrumentationDiv')
     .contains('On');
   cy.get('@instrumentationDiv')
     .find('.toggle-switch')
     .click();
   cy.get('@instrumentationDiv')
     .contains('Off');
   cy.get('[data-testid=config-apply-close]')
     .click();

   // Name pipeline then deploy pipeline
   cy.get('.pipeline-name')
     .click();
   cy.get('#pipeline-name-input')
     .type(TEST_PIPELINE_NAME)
     .type('{enter}');
   cy.get('[data-testid=deploy-pipeline]')
     .click();

   // Do assertions
   cy.url()
     .should('include', '/view/__UI_test_pipeline');
   cy.contains(TEST_PIPELINE_NAME);
   cy.contains('FileDelete');
   cy.contains('Configure')
     .click();
   cy.contains('Pipeline config')
     .click();
   cy.get('.label-with-toggle')
     .contains('Instrumentation')
     .parent()
     .contains('Off');
   cy.get('[data-testid=close-modeless]')
     .click();

   // Delete the pipeline to clean up
   cy.get('.pipeline-actions-popper')
     .click();
   cy.get('[data-testid=delete-pipeline]')
     .click();
   cy.get('[data-testid=confirmation-modal]')
     .find('.btn-primary')
     .click();

   // Assert pipeline no longer exists
   cy.contains(TEST_PIPELINE_NAME)
     .should('not.exist');
 });
});

When running the above test in Cypress test runner, Cypress will actually create a browser instance, and go through each step. However, it will be slow to have all of our tests actually running in UI. Cypress provides the ability to make API requests, so we can also do API calls to set up data instead of manually creating through UI. For even faster tests, we can stub data in Cypress as well.

For performance reasons, we should only have one true end-to-end test for a feature (e.g. test the ‘happy path’), and have stubbed responses for the rest of the tests.

CI (Bamboo integration)

We can set up a plan on Bamboo, that will run our entire UI test suite whenever a new commit is pushed to a branch. The UI developer should make sure that there’s a green build before merging their PR, to make sure there are no regressions when adding new features.

When we integrate Cypress with Bamboo, Cypress can record the tests, and we can see the results in Cypress’ Dashboard service. This is a very cool feature that not only records and takes screenshots of the tests runs in CI, but it also automatically parallelizes and load balances so that tests run much faster. This service is free for up to 3 users.

Test coverage (future)

It is not crucial at the beginning when we are just starting to write unit tests, but in the future we should keep track off and try to improve test coverage over time. We should also make sure that the test coverage for the codebase doesn’t decrease as we build more features. There are several tools that we can integrate with to help with this, e.g. Coveralls.