UI testing with Puppeteer and Gitlab CI

Bill Gates once said, “Programmers should be lazy enough to find simple solutions to difficult tasks.” One of the most challenging tasks is certainly the regulation of a website that contains several subpages in 8 resolutions and is available in 3 language variations. Do you agree?

At one of our regular internal knowledge and skills sharing meetings, we also addressed the topic of user interface (UI) testing in web applications. One of my colleagues came up with an overview of the currently available solutions. During the presentation, another colleague got excited when hearing about Puppeteer (from the Chrome browser developers) and immediately started experimenting.

Puppeteer is a so-called headless browser, i.e. a web browser that has no graphical user interface. So it's some kind of software that accesses websites but doesn't display them

Puppeteer, however, has its own innumerable problems, which is why my colleague’s first attempt was a failure. Puppeteer resisted being installed from Dockerfile, constantly freezing and being “down”. But then I came up with a lazy idea – is there any Docker image already on the Docker Hub? And of course there was.

After playing around for a while, Docker image I managed to run the “Hello World” app from our Gitlab CI to have a two-step pipeline when deploying. After my first successful deployment, I set this step to be skipped. You know, an extra 30 seconds to live. In this step, however, you can deploy your website, which you want to subsequently test.

stages:
 - build
 - test

build:
  stage: build
  image: node:carbon
  when: manual
  script:
   - cd app
   - npm install
  tags:
    - docker

test:
  stage: test
  image: alekzonder/puppeteer:1.5.0-0
  script:
   - cd test
   - yarn install
   - node screenshot-test.js
  artifacts:
    paths:
     - test/screenshots/*/*
    expire_in: 3 days
  tags:
    - docker

When we look at the “gitlab-ci.yml” file – the “build” step installs the Node application and the “test” step starts the UI test script written in JavaScript.

The “script” contains the commands to be executed. We enter the “test” folder, which contains the test script and a definition of the necessary third-party libraries that we install using Yarn, and finally the test itself launches using Node.js.

Artifacts are files presented by the CI chain – in our case, they are individual screenshots. They can be set for a limited period of time – for example, you can set them to 3 days, which means that after this time, they will be deleted from the server and free up space.

The “docker” tag launches our Gitlab Runner supporting Docker.

{
  "name": "hello-test",
  "scripts": {
    "start": "node screenshot-test.js"
  },
  "private": true,
  "dependencies": {
    "chromeless": "^1.5.0",
    "puppeteer": "1.5.0"
  }
}

The “package.json” file (listed above) defines the boot command and contains a request for a puppeteer package that further requires the download and installation of the Chromium browser.

Ever since version 1.7.0 is available, it is possible to use the puppeteer-core package and use your locally or remotely installed Chromium.

const puppeteer = require('puppeteer');
const fs = require('fs');
/**
 * Launcher
 */
(async () => {
  //
  // set up Puppeteer
  //
  let browser;
  try {
    browser = await puppeteer.launch({
      args: [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-dev-shm-usage'
      ]
    });
  } catch (err) {
    console.log('Error launching browser');
    throw err;
  }

  // launch browser
  try {
    fs.mkdirSync('screenshots/' + name);
    // Web page
    const page = await browser.newPage();
    await page.setViewport({
      width: 1920,
      height: 1080
    });
    await page.goto('https://example.com/', {
      timeout: 10000,
      waitUntil: 'networkidle0'
    });

    // full page screenshot
    const bodyHandle = await page.$('body');
    const boundingBox = await bodyHandle.boundingBox();

    await page.screenshot({
      path: 'screenshots/example-com-fullpage.jpg',
      clip: {
        x: 0,
        y: 0,
        width: parseInt(boundingBox.width),
        height: parseInt(boundingBox.height)
      },
      type: 'jpeg'
    });

    await bodyHandle.dispose();

The simple script that we run in the “screenshot-test.js” file makes a screenshot of the loaded website.

The first thing we need to do is open the browser on the computer. To prevent memory constraints on “high” pages, we enter the “disable-dev-shm-usage” command.

Secondly, we need to open a new tab in the browser, set the target resolution, and then we just go to the desired page. To make sure it loads correctly, we set the timeout to 10,000 ms and wait for the network to become inactive and all loads to end.

Since we want to create a snapshot of the entire website, we need to “play with it” a little bit (hack) and pull its dimensions in the form of height and width of the screenshot. At this point, there may be problems with extremely high websites, but you'll have an idea of what the site looks like.

After that, you can just close the entire browser. However, what if we want to test more websites or resolutions? Do we need to close each page or browser one by one? No, of course not. But on the other hand, it pays off because you want to load the page after some change in resolution is made. This ensures that all CSS styles are correct.

await page.reload({ timeout: 10000, waitUntil: 'networkidle0' });

What else can you do with Puppeteer? It's more or less up to you. Please note that you have a browser with a large API that can be used in Node.js. Sounds good, doesn't it?

Let's see how you would test your server's “status code” response:

const assert = require('assert');
const STATUS_OK = 200;
try {
  let response = await page.goto(url, {
    timeout: 10000,
    waitUntil: 'networkidle0'
  });
} catch (err) {
  return Promise.reject(err);
}
if (!response) {
  return Promise.reject(new Error('Response empty'));
}
assert.equal(response.status(), STATUS_OK, 'Wrong status code');

Or how would you test specific elements on the website:

// Link hostname only
let result = await page.$eval('a#sk', link => link.hostname);
assert.equal(result, 'example.sk');
// Link full URL
result = await page.$eval('a#sk', link => link.href);
assert.equal(result, 'https://example.sk/link');
// Element content
result = await page.$eval('h1', link => link.innerHTML);
assert.equal(result, 'Main header');