Soak testing with Playwright

The short version

The package repeats a flow you give it a few hundred times in a single browser session, and measures the DOM node and event listener counts as it goes. If those counts keep climbing, the test fails.

• Repeats browser interactions against a live application
• Keeps the same browser, page, and application session open for the whole run
• Runs a lot of passes quickly
• Installs Playwright's virtual clock and advances it, so code that runs on timers runs without the test having to wait
• Measures DOM nodes, event listeners, heap size, and the number of documents after forcing garbage collection
• Fits a trend line across the run, so it can tell growth that keeps going apart from a single jump
• Turns what happens to a page over a long session into an automated regression test

It expects a flow that ends where it started, like opening a drawer and closing it, or filtering a table and clearing the filter, so the counts at the end of each pass should match the counts at the beginning. A flow that adds to the page on purpose, like an infinite scroll feed, is going to grow whatever happens, so it's a better fit for the measurement function than for the assertion.

Installation

npm install --save-dev playwright-soak-test

@playwright/test is a peer dependency, so the package uses the Playwright version you already have, which needs to be v1.45 or newer. The counts come from the Chrome DevTools Protocol, so soak tests are skipped on Firefox and WebKit.

A first soak test

You import test from the package instead of from Playwright, and put the part you want repeated inside soak.run(). Everything outside it, like navigation, login, and setup, only runs once.

import { test } from 'playwright-soak-test';

test('the dashboard drawer does not leak memory', async ({ page, soak }) => {
  await page.goto('/dashboard');

  await soak.run(async () => {
    await page.getByRole('button', { name: 'Report' }).click();
    await page.getByRole('button', { name: 'Close' }).click();
  });
});

That flow runs 200 times. A baseline reading is taken after 5 warmup passes, and if a count has grown past its threshold by the end, soak.run() throws and the test fails.

If you'd rather have the measurements than an assertion, for example to find out how much your application grows before you set a budget, soak.measure() takes the same arguments and returns the result instead of throwing.

const result = await soak.measure(openAndCloseDrawer);

console.log(result.trends.nodes.perPass);   // e.g. 40
console.log(result.trends.listeners.total); // e.g. 195
console.log(result.leaking);                // true

Project configuration

A 200 pass run is better kept apart from your ordinary end to end tests. It works best in its own Playwright project, matched by filename, with those files excluded from the projects that run on every commit.

import { defineConfig, devices } from '@playwright/test';
import { soakLaunchOptions } from 'playwright-soak-test';

export default defineConfig({
  reporter: [
    ['list'],
    ['playwright-soak-test/reporter'],
  ],
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      testIgnore: /.*\.soak\.spec\.ts/,
    },
    {
      name: 'soak',
      testMatch: /.*\.soak\.spec\.ts/,
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: soakLaunchOptions,
        // Both of these keep large buffers in the renderer, which affects readings
        trace: 'off',
        video: 'off',
      },
    },
  ],
});

soakLaunchOptions includes --js-flags=--expose-gc, which lets the package reliably force garbage collection before each reading. Then you can run the soak project by itself:

npx playwright test --project=soak

The measurements

Before each reading, the package forces garbage collection and then gets four values from Chromium, so memory that the browser was about to free doesn't get counted as retained.

Instead of taking a reading on every pass, the run takes one after each of the first 25 passes, then one every Nth pass after that, and fits a straight line through the samples. The slope of that line is the per pass figure in the report, and how closely the samples follow the line is how the package tells a leak apart from a one off jump.

The virtual clock

The fixture installs Playwright's virtual clock before the application loads, and pauses it once the application has started. After that, the clock only advances when a pass advances it, by advanceMs each time. That value works best when it matches the interval your application actually uses.

For a poller that calls an endpoint every 30 seconds, that's 30 seconds a pass. If you fulfill the request yourself, the pass doesn't go out to the network, and if you tell the pass which response to wait for, it waits for the request that the clock advance triggered:

test.use({
  soakOptions: {
    clock: {
      advanceMs: 30_000
    }
  }
});

test('the dashboard drawer does not leak', async ({ page, soak }) => {
  await page.route('**/api/feed', route => route.fulfill({ json: feed }));
  await page.goto('/dashboard');

  await soak.run(openAndCloseDrawer, {
    waitForResponse: '**/api/feed'
  });
});

The package starts listening for waitForResponse before the flow runs and waits for it after the clock advances, so it still catches a response that the advance triggered. If the wait times out, it adds to the responseTimeouts count on the result and the pass continues. For WebSockets, page.routeWebSocket() does the same job on Playwright 1.48 and above.

Options

You can set defaults in use: { soakOptions } in the config, or at the top of a spec file with test.use. Any option can also be passed for a single run, as the second argument to soak.run and soak.measure.

Read the report

A failing run prints what grew, by how much, whether it grew steadily or in one jump, and where to look next.

Memory leak detected in "the dashboard drawer does not leak".

Listeners     +195  (threshold 0)    ▁▁▁▁▁▁▁▂▂▂▂▂▂▂▃▃▄▅▅▆▆▇██  +1.0 per pass, R²=1.00

DOM nodes   +7,800  (threshold 100)  ▁▁▁▁▁▁▁▂▂▂▂▂▂▂▃▃▄▅▅▆▆▇██  +40.0 per pass, R²=1.00

Heap       +10.37%  (reported only)  ▁▁▁▂▂▂▂▂▂▂▂▃▃▃▃▄▅▅▆▆▇▇██  1.03 MB → 1.13 MB

Every pass leaks 40.0 nodes and 1.0 listeners, starting from the first one.

200 passes x 18s of virtual time = 1h of app time.

The per pass figure is how much one run of the journey adds, and that's usually the figure to quote when you file the bug. An R² near 1 means the samples fall on a straight line, which is what a real leak looks like.

If the growth stopped partway through, it's reported as over the threshold rather than as a leak, and a single jump is labeled with the pass where it happened. That way you can tell a leak from a chunk that was lazily loaded without having to work it out yourself. The reporter prints a box for each test and a table at the end of the run, and on GitHub Actions it also writes an error annotation and a job summary.

API

Each call returns a SoakResult containing the baseline, the final reading, all the samples taken, and a fitted trend for each metric with its slope, total, fit, and shape. SoakLeakError includes the same object on its result property, so you still have all of the data to work from when a run fails.

{
  label: 'the dashboard drawer does not leak',
  passes: 200,
  warmup: 5,
  baseline: { heap: 1079204, nodes: 258, listeners: 22, documents: 1 },
  after:    { heap: 1190724, nodes: 8058, listeners: 217, documents: 1 },
  trends: {
    nodes:     { perPass: 40, total: 7800, shape: 'linear', r2: 1 },
    listeners: { perPass: 1,  total: 195,  shape: 'linear', r2: 1 },
    heap:      { perPass: 544.57, total: 111520, shape: 'linear', r2: 0.99 },
  },
  leaking: true,
  clock: { enabled: true, advanceMs: 18000, virtualElapsedMs: 3610000 },
  responseTimeouts: 0,
}

The possible values for shape are flat, linear, step, settled, and noisy, and the package works them out from how well the samples fit the line. They're the same categories that the trajectory analysis uses in the methodology.

In continuous integration

A 200 pass run finishes in seconds, so a soak check can run in the same pipeline as everything else. It works best as its own job, with one worker and no retries, so each run has the browser to itself:

- name: Soak test
  run: npx playwright test --project=soak --workers=1 --retries=0

For longer runs, you'll need to raise Playwright's own test timeout, which defaults to 30 seconds:

test.setTimeout(2 * 60 * 60 * 1000);
test.use({ soakOptions: { passes: 10_000 } });

A long run reports its progress as it goes, so you can follow the log of a scheduled job while it's still running:

[playwright-soak-test] the dashboard drawer does not leak: 4,000/10,000 passes, 12m, nodes +0, listeners +1

Limitations

There are a few things to know before you read too much into a result:

• Chromium only. Soak tests are skipped on Firefox and WebKit, because the counts come from the DevTools Protocol.
• Readings vary between runs. That's why soak runs work best in their own job, with one worker and no retries.
• Clicking an element that gets removed can add nodes. In Chromium, clicking an element that the flow then removes adds a couple of retained nodes per pass, but only when the application is already keeping that subtree in memory. A build that doesn't leak still measures zero growth.
• Pseudo elements count. A ::before or ::after with generated content adds a pseudo element and its text to the node count.
• Node and listener counts don't include data kept outside the DOM. A poller that keeps every response in an array can make the heap grow a lot while the node and listener counts don't change at all. heapThresholdPercent catches that case.