Automatically detect memory leaks with Puppeteer

About half a year ago Bronley Plumb kindly made me aware of a memory leak in one of my Open Source packages. To see this memory leak in action it was necessary to open a browser and its dev tools to execute some manual steps. On top of that, the memory had to be inspected manually. It was a complicated procedure. Normally I just add a failing test before I fix a bug. This time it was a bit more tricky. But in the end I found a way to test the memory consumption automatically and here is what I came up with.

If you are not interested in the adventurous path which led me to the solution feel free to skip right to the end to read on from there.

What is a memory leak?

In general a memory leak is the situation in which a software holds on to a piece of memory which it doesn't really need anymore. In JavaScript this most likely means that there is a reference to an object somewhere which you totally forgot about. But for the garbage collection it is impossible to distinguish between objects which are still in use and those that have just been forgotten somewhere.

Historically a memory leak was something that web developers did not have to care about. Every link on a page caused a new page to be loaded which in turn wiped the memory. But memory leaks are usually very shy and only become noticeable when a particular program keeps running for a long time.

With todays Single Page Applications and Progressive Web Apps the situation has changed. Many websites do behave like apps and are designed to run for a long time and that is particularly true for apps that use the Web Audio API. The memory leak in question was found in standardized-audio-context which is a library to achieve cross browser compatibility for that API.

The most simple example of a memory leak that I could think of is attaching some metadata to an object. Let's say you have a couple of objects and you want to store some metadata for each of them. But you don't want to set a property on those objects because you want to keep the metadata in a separate place.

This can be solved by using a Map as shown in the following snippet. It allows to store some metadata, to get it back and to delete it again. All that is needed is a Map which uses an object as the key to index its metadata.

const map = new Map();

// store metadata
map.set(obj, metadata);

// get metadata
map.get(obj);

// delete metadata
map.delete(obj);

But what if an object with metadata is not referenced anywhere else anymore? It still can't be garbage collected because the Map still has a reference to it to index the metadata.

The following example is of course contrived but many memory leaks can be reduced to something as simple as this.

const map = new Map();

setInterval(() => {
    const obj = { };

    map.set(obj, { any: 'metadata' });
}, 100);

All the created objects do survive every garbage collection because the Map still has a reference to them. This is the perfect use case for a WeakMap. The references held by a WeakMap do not prevent any object from being garbage collected.

const map = new WeakMap();

setInterval(() => {
    const obj = { };

    map.set(obj, { any: 'metadata' });
}, 100);

By replacing the Map with a WeakMap this common cause for a memory leak can be eliminated. The problem that caused the memory leak in my code was very similar although it was not that obvious to spot.

What is puppeteer?

Puppeteer is a tool which can be used to remote control Chrome or any other Chromium browser. It is a simpler alternative to Selenium and WebDriver but it has the downside that it only works with browsers based on Chromium (for now). It comes with access to some APIs which are not accessible by Selenium because it tries to interact with a website like a real user. Puppeteer on the other hand has access to many APIs which are not accessible to normal users. This works by utilizing the Chrome DevTools Protocol. One of those things that Puppeteer can do which Selenium can't is inspecting the memory. And this is of course super helpful when trying to find memory leaks.

Measuring the memory usage

At first glance there seems to be a function in the API of Puppeteer which offers all that is needed to track the memory usage. It's the page.metrics() method. It does among other things also return a metric called JSHeapUsedSize. This is the number of bytes that V8, the JavaScript engine used in Chrome, uses as memory.

const { JSHeapUsedSize } = await page.metrics();

Triggering the garbage collection

Unfortunately getting the size of the memory is not enough. The memory of a JavaScript program is managed by a very autonomous garbage collection. Unlike the garbage collection in the real world which usually shows up on a very strict and well known schedule the JavaScript garbage collection does its job whenever it thinks it's the right time to do so. It can normally not be triggered from within the JavaScript code. But it is necessary to make sure it ran before inspecting the memory to be sure that all the trash has been picked up and the memory consumption has been computed based on the latest changes made by the code.

However Puppeteer (at least at version 1.17) has no dedicated method to trigger the garbage collection. But according to this comment on a GitHub issue it should be fairly easy to implement. The author suggests to use the internal DevTools client to interact with the HeapProfiler directly.

await page._client.send('HeapProfiler.enable');
await page._client.send('HeapProfiler.collectGarbage');
await page._client.send('HeapProfiler.disable');

But as signaled by the underscore prefix, the _client property is not part of Puppeteer's API. It may be changed or even removed at any time. Using it is more of a hack then a robust solution.

Besides it being a fragile hack I was also not able to get consistent results when executing the three commands above. I therefore tried a different way to trigger the garbage collection.

When taking a snapshot of the heap the garbage collection will be triggered implicitly. And that's why I tried to programmatically create a snapshot as shown in the following snippet.

await new Promise((resolve) => {
    const resolvePromise = ({ finished }) => {
        if (finished) {
            page._client.off(
                'HeapProfiler.reportHeapSnapshotProgress',
                resolvePromise
            );

            resolve();
        }
    };

    page._client.on(
        'HeapProfiler.reportHeapSnapshotProgress',
        resolvePromise
    );
    page._client.send(
        'HeapProfiler.takeHeapSnapshot',
        { reportProgress: true }
    );
});

Again it didn't work. It did for sure call the garbage collection but it wasn't producing consistent results as expected.

I went on to try something different. According to some Stack Overflow answers it should be possible to specify V8 flags when launching Puppeteer. One of those flags is meant to tell it to expose the garbage collection.

await puppeteer.launch({
    args: [ '--js-flags=--expose-gc' ]
});

Then it is possible to call a magic gc() function which is attached to the global window object.

await page.evaluate(() => gc());

This feels less hacky then using the internal DevTools Protocol client of Puppeteer but again I wasn't able to get consistent results.

Counting all the objects

I wondered why the results of my tests varied so much. Even when using the most simple code snippets the memory usage was different when I changed one parameter of my test case.

After some time it occured to me that the memory usage seems to depend on the number of iterations that I ran. To make sure that I don't have a memory leak I executed a piece of code several times. My expectation was that the memory consumption should remain unchanged. Or in other words there should be nothing left in the memory when the piece of code I was testing ran to completion.

But there was some relationship between the memory usage and the number of iterations. However it was not a linear relationship as one would expect in case of an actual memory leak. For some numbers the memory increased, for others it remained almost stable and for even others it shrinked!

I concluded that I might be testing something that is at least one level to deep. V8 as well as every other JavaScript engine is very complex. It tries to optimize the code which it has to execute as much a possible. The V8 team has for example a blog where they post regular articles about new optimizations that they applied.

If you create an object in JavaScript there is no guarantee on how much memory it will be using. A browser (or its JavaScript engine) may choose to store it in an extremely memory efficient way first, but when you use it heavily it might decide to switch to an alternative version which consumes more space on your machine but is much faster to access. Who knows?

I think this was exactly the problem that I ran into. The memory consumption changed dependending on the number of iterations I ran to make sure that there is no unused memory left over. V8 sometimes optimized the code in different ways which did in turn change its memory footprint. It would have been an option to study the internals of V8 to compensate for its optimizations when measuring the memory. But these optimizations get refined regularly and it would make my tests very brittle to rely on that.

There is a better and even simpler way. Puppeteer offers a method to query all objects with a given prototype. It's called page.queryObjects().

As shown in the following function it takes a prototype and counts all objects which have the same prototype somewhere in there prototype chain. In this case I used the Object.prototype because in JavaScript almost everything inherits from it. Notable exceptions to that rule are objects which have been created without a prototype (by using Object.create(null)) and all primitive values.

const countObjects = async (page) => {
    const prototype = await page.evaluateHandle(() => {
        return Object.prototype;
    });
    const instances = await page.queryObjects(
        prototype
    );
    const numberOfObjects = await page.evaluate(
        (instances) => instances.length,
        handle
    );

    await prototype.dispose();
    await instances.dispose();

    return numberOfObjects;
};

This technique is independent of the number of bytes these objects occupy on the heap. It just counts their number and I think this is at least a good start to use it to check JavaScript code for memory leaks.

Finally I found a way to get consistent results. And the best thing about it is that it will also trigger the garbage collection internally before counting the objects. That way I don't have to use one of the hacky solutions mentioned above.

Joining the forces

In theory it should be enough to only use the countObjects() function defined above. But unfortunately it has a subtle bug at the moment which I wouldn't have been able to find without the very helpful input of Andrey Lushnikov. Many thanks for that!

It turns out that functions which return an array or an object trick it to keep a copy of the returned value in memory. But it is possible to use the very first function to measure the memory that I tried along the way to build a workaround for that issue.

As already said the countObjects() function gets used to count all the instances of objects which inherit from the Object.prototype. But right after that the heap size gets queried as well. This is done by using the page.metrics() method. It's not really done to get the actual number of used bytes. The purpose of calling it is to determine when subsequent garbage collection runs do not have an effect on the heap size anymore.

As long as the heap size keeps decreasing we keep on calling the function recursively. Once the heap size keeps the same for two consecutive runs the number of objects is assumed to be correct and gets returned.

const countUntilStable = async (
    page,
    previousHeapSize = null
) => {
    const numberOfObjects = await countObjects(page);
    const { JSHeapUsedSize } = await page.metrics();

    if (previousHeapSize === JSHeapUsedSize) {
        return numberOfObjects;
    }

    return countUntilStable(page, usedHeapSize);
};

And now we can use this countUntilStable() function inside of the memory leak tests like this.

it('should not have a memory leak', async () => {
    const numberOfObjects = await countUntilStable(page);

    await page.evaluate(() => {
        // Do something a couple of times.
    });

    expect(await countUntilStable(page))
        .to.equal(numberOfObjects);
});

And with that we finally have a solution to test for memory leaks in an automated way.

In case you are interested feel free to explore the memory leak tests I wrote for the standardized-audio-context library to make sure the reported memory leak never comes back.

If you encounter a bug in any of my Open Source projects please report it. I often start projects as part of my work for clients. But the maintenance is usually done in my free time. This is the reason why it can sometimes take a little longer to fix bugs. But even if it takes me half a year I'm comitted to fix all the bugs you find.