Soak testing with Playwright

The short version

The package repeats a flow you provide a few hundred times inside a single browser session, watching DOM node and event listener counts as it goes. If they keep climbing, the test fails.

• Repeatedly executes browser interactions against a live application
• Keeps the same browser, page, and application session alive throughout
• Runs large numbers of iterations quickly
• Installs and drives Playwright's virtual clock, so timer driven behavior happens without waiting
• Measures DOM nodes, event listeners, heap size, and document count after a forced collection
• Fits a trend across the run and identifies sustained growth, distinguishing it from a single step
• Turns long lived frontend behavior into an automated regression test

It expects a flow that ends where it started, like opening a drawer and closing it, or filtering a table and clearing the filter, so the counts at the end should match the counts at the beginning. A flow that adds to the page on purpose, like an infinite scroll feed, grows no matter what, and suits the measurement function better than the assertion.

Installation

npm install --save-dev playwright-soak-test

@playwright/test is a peer dependency, so it uses the Playwright version you already have, and needs at least v1.45. The counts come from the Chrome DevTools Protocol, so the fixture skips itself on Firefox and WebKit.

A first soak test

test comes from the package instead of from Playwright, and the part you want repeated goes inside soak.run(). Everything outside the wrapper, like navigation, login, and setup, happens once.

import { test } from 'playwright-soak-test';

test('the dashboard drawer does not leak memory', async ({ page, soak }) => {
  await page.goto('/dashboard');

  await soak.run(async () => {
    await page.getByRole('button', { name: 'Report' }).click();
    await page.getByRole('button', { name: 'Close' }).click();
  });
});

That flow runs 200 times. A baseline gets taken after 5 warmup passes, and if a count has grown past its threshold by the end, soak.run() throws and the test fails.

When you want the numbers rather than an assertion, to see where an application stands before setting any budget, soak.measure() takes the same arguments and returns the result instead of throwing.

const result = await soak.measure(openAndCloseDrawer);

console.log(result.trends.nodes.perPass);   // e.g. 40
console.log(result.trends.listeners.total); // e.g. 195
console.log(result.leaking);                // true

Project configuration

A 200 pass run doesn't belong alongside your ordinary end to end tests. It wants its own project, matched by filename, with those files excluded from the projects that run on every commit.

import { defineConfig, devices } from '@playwright/test';
import { soakLaunchOptions } from 'playwright-soak-test';

export default defineConfig({
  reporter: [
    ['list'],
    ['playwright-soak-test/reporter'],
  ],
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      testIgnore: /.*\.soak\.spec\.ts/,
    },
    {
      name: 'soak',
      testMatch: /.*\.soak\.spec\.ts/,
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: soakLaunchOptions,
        // Both of these keep large buffers in the renderer, which affects readings
        trace: 'off',
        video: 'off',
      },
    },
  ],
});

soakLaunchOptions carries --js-flags=--expose-gc, which is what makes the forced collection before each reading reliable. The soak project then runs on its own:

npx playwright test --project=soak

The measurements

Every reading takes four values from Chromium after forcing garbage collection, so memory the browser was about to release doesn't get counted as retained.

Rather than reading on every pass, the run reads each of the first 25 individually and then every Nth pass afterwards, fitting a line through the samples. The slope of that line is the per pass figure in the report, and how well the samples fit it separates a leak from a one off step.

The virtual clock

The fixture installs Playwright's virtual clock before the application loads, and pauses it once the application is up. From then on time only moves when a pass moves it, by advanceMs each time, which wants matching to the interval your application actually uses.

For a poller that calls an endpoint every 30 seconds, that's 30 seconds a pass. Fulfilling the request yourself keeps the pass off the network too, and naming the response makes the pass wait for the work the advance triggered:

test.use({
  soakOptions: {
    clock: {
      advanceMs: 30_000
    }
  }
});

test('the dashboard drawer does not leak', async ({ page, soak }) => {
  await page.route('**/api/feed', route => route.fulfill({ json: feed }));
  await page.goto('/dashboard');

  await soak.run(openAndCloseDrawer, {
    waitForResponse: '**/api/feed'
  });
});

waitForResponse gets registered before the flow runs and awaited after the clock moves, so a response triggered by the advance is still caught. A wait that times out adds to responseTimeouts on the result, and the pass carries on. For sockets, page.routeWebSocket() does the same job on Playwright 1.48 and above.

Options

Defaults go in use: { soakOptions } in the config, or at the top of a spec with test.use. Any of them can also be passed per run, as the second argument to soak.run and soak.measure.

Read the report

A failing run prints what grew, by how much, the shape of the growth, and where to look next.

Memory leak detected in "the dashboard drawer does not leak".

Listeners     +195  (threshold 0)    ▁▁▁▁▁▁▁▂▂▂▂▂▂▂▃▃▄▅▅▆▆▇██  +1.0 per pass, R²=1.00

DOM nodes   +7,800  (threshold 100)  ▁▁▁▁▁▁▁▂▂▂▂▂▂▂▃▃▄▅▅▆▆▇██  +40.0 per pass, R²=1.00

Heap       +10.37%  (reported only)  ▁▁▁▂▂▂▂▂▂▂▂▃▃▃▃▄▅▅▆▆▇▇██  1.03 MB → 1.13 MB

Every pass leaks 40.0 nodes and 1.0 listeners, starting from the first one.

200 passes x 18s of virtual time = 1h of app time.

The per pass figure is what one run of the journey adds, which is usually the number worth quoting when you file the bug. An R² near 1 means the samples sit on a straight line, which is what a real leak looks like.

Growth that stopped gets reported as over threshold rather than as a leak, and a single jump gets labeled with the pass it landed on. That's the difference between a leak and a lazily loaded chunk, worked out for you. The reporter prints a box per test and a table at the end of the run, and on GitHub Actions it also writes an error annotation and a job summary.

API

Every call returns a SoakResult carrying the baseline, the final reading, every sample taken, and a fitted trend per metric with its slope, total, fit, and shape. SoakLeakError carries the same object on its result property, so a failing run gives you the full data set to work from.

{
  label: 'the dashboard drawer does not leak',
  passes: 200,
  warmup: 5,
  baseline: { heap: 1079204, nodes: 258, listeners: 22, documents: 1 },
  after:    { heap: 1190724, nodes: 8058, listeners: 217, documents: 1 },
  trends: {
    nodes:     { perPass: 40, total: 7800, shape: 'linear', r2: 1 },
    listeners: { perPass: 1,  total: 195,  shape: 'linear', r2: 1 },
    heap:      { perPass: 544.57, total: 111520, shape: 'linear', r2: 0.99 },
  },
  leaking: true,
  clock: { enabled: true, advanceMs: 18000, virtualElapsedMs: 3610000 },
  responseTimeouts: 0,
}

The shapes are flat, linear, step, settled, and noisy, worked out from how well the samples fit the line. They map directly onto the trajectory analysis in the methodology.

In continuous integration

Because a 200 pass run finishes in seconds, a soak check can sit in the same pipeline as everything else. It wants a dedicated job with one worker and no retries, so each run gets a browser to itself:

- name: Soak test
  run: npx playwright test --project=soak --workers=1 --retries=0

Longer runs need Playwright's own test timeout raised, since it defaults to 30 seconds:

test.setTimeout(2 * 60 * 60 * 1000);
test.use({ soakOptions: { passes: 10_000 } });

A long run reports progress as it goes, so a scheduled job stays readable for the whole hour:

[playwright-soak-test] the dashboard drawer does not leak: 4,000/10,000 passes, 12m, nodes +0, listeners +1

Limitations

Worth knowing before you read too much into a result:

• Chromium only. The counts come from the DevTools Protocol, so the fixture skips itself on Firefox and WebKit.
• Readings vary between runs. Soak runs want their own job, one worker, no retries.
• Clicking a removed element leaves a trace. Clicking an element the flow then removes adds a couple of retained nodes per pass in Chromium, but only on a subtree the application is already keeping. A clean build still reads zero.
• Pseudo elements count. A ::before or ::after with generated content adds a pseudo element and its text to the node count.
• Counts miss what stays out of the DOM. A poller that keeps every response in an array can grow the heap substantially with node and listener counts dead flat. heapThresholdPercent is what catches that case.