Using Netlify Edge Functions for request rewrites to run page level A/B tests on a Gatsby site

We recently started trying out Netlify’s Edge Functions for A/B testing and we were pleasantly surprised that without much effort we were able to get it working with our existing Gatsby site. Although setting the test up required quite a dirty patch to Gatsby core, once that was worked out it’s mostly pain free.

Here’s the demo repo in case you’re interested in setting this up for yourself. We’ll dig into the code below.

An important caveat to keep in mind: whilst page level A/B testing is sometimes your only option, where possible you should consider doing your A/B testing client-side. For changes that affect content above the fold, or just in general can’t wait for JavaScript to load, you’ll usually reach for page level A/B testing. However, for content which is already behind a loading spinner you’re better off doing it all client-side to avoid making your code unnecessarily complicated.

The naive approach

Let’s start with the simplest possible configuration. Two Gatsby pages which live at /home-a/ and /home-b/ which we would like to switch between depending on a cookie value in the Netlify Edge Function.

// netlify/edge-functions/abtest.ts

import type { Context } from "https://edge.netlify.com";

export default (request: Request, context: Context) => {
  // look for existing "test_bucket" cookie
  const bucketName = "test_bucket";

  let bucket = context.cookies.get(bucketName);

  if (!bucket) {
    // if no "test_bucket" cookie is found, assign the user to a bucket
    // in this example we're using two buckets (a, b) with an equal weighting of 50/50
    const weighting = 0.5;

    // get a random number between (0-1)
    const random = Math.random();
    bucket = random <= weighting ? "a" : "b";

    // set the new "test_bucket" cookie
    context.cookies.set({
      name: bucketName,
      value: bucket,
      maxAge: 365 * 24 * 60 * 60,
    });
  }

  const rewritePage = bucket === 'a' ? '/home-a/' : '/home-b/'
  
  return context.rewrite(rewritePage)
};

Great, now we can switch between /home-a/ and /home-b/ depending on cookie. But how will Gatsby handle this confusing setup? Two possible pages it needs to render at the same path depending on the value of a cookie? Let’s see:

// gatsby-node.ts

import path from 'path'
import { GatsbyNode } from "gatsby";

export const createPages: GatsbyNode['createPages'] = ({ actions }) => {
  actions.createPage({
    component: path.resolve('./src/templates/home-a.tsx'),
    path: '/home-a',
  })
  actions.createPage({
    component: path.resolve('./src/templates/home-b.tsx'),
    path: '/home-b',
  })
}

If we run this everything will work fine until JavaScript loads, at which point Gatsby’s client-side router will take over, decide that you actually meant to be at /home-b/ or /home-a/ instead of / and promptly updates the URL. So it’s working… but having the URL change kind of defeats the purpose of doing an edge rewrite. See below:

For the A test:

And for the B test:

But disabling JavaScript we get what we want, no redirecting:

One step closer

How do we fix this? Well let’s tell Gatsby that it’s OK to have /home-b/ and /home-a/ both live at / by using the matchPath option in the createPage API call.

// gatsby-node.ts

import path from 'path'
import { GatsbyNode } from "gatsby";

export const createPages: GatsbyNode['createPages'] = ({ actions }) => {
  actions.createPage({
    component: path.resolve('./src/templates/home-a.tsx'),
    path: '/home-a',
		matchPath: '/', // <---
  })
  actions.createPage({
    component: path.resolve('./src/templates/home-b.tsx'),
    path: '/home-b',
		matchPath: '/', // <---
  })
}

This should work now right? Gatsby knows that we want these pages to live at / and it should be fine with either /home-a/ or /home-b/ showing up? Well if you think about it from Gatsby’s data loader perspective, which page data and page template should be loaded here? Both /home-a/ and /home-b/ template are valid options, and so we have ambiguity.

Gatsby is just going to pick one of the two pages and always render that and we’re in the same position we were before adding matchPath . In this case, it always picks the first page created using createPage which will be /home-a/ . So we’ll stay on / but we will always get the A test, regardless of cookie value. You’ll see a flash of the B variant from the server HTML if you’re assigned that cookie by the Edge Function but will be swiftly redirected. See below:

A necessary evil (aka. patching node_modules)

Where is this happening? What is doing this client-side redirect? Our site works fine before JavaScript loads! Maybe we can dig into Gatsby to find this JavaScript and modify it so it understands our use case (or ideally interact with it using an API)? If we look at the cache-dir/find-path.js file in Gatsby we’ll see that it’s making a decision based on matchPath values:

// Given a raw URL path, returns the cleaned version of it (trim off
// `#` and query params), or if it matches an entry in
// `match-paths.json`, its matched path is returned
//
// E.g. `/foo?bar=far` => `/foo`
//
// Or if `match-paths.json` contains `{ "/foo*": "/page1", ...}`, then
// `/foo?bar=far` => `/page1`
export const findPath = rawPathname => {
  const trimmedPathname = trimPathname(absolutify(rawPathname))
  if (pathCache.has(trimmedPathname)) {
    return pathCache.get(trimmedPathname)
  }

  const redirect = maybeGetBrowserRedirect(rawPathname)
  if (redirect) {
    return findPath(redirect.toPath)
  }

  let foundPath = findMatchPath(trimmedPathname)

  if (!foundPath) {
    foundPath = cleanPath(rawPathname)
  }

  pathCache.set(trimmedPathname, foundPath)

  return foundPath
}

If we can change this to tell it that when you load / we actually want to load /home-a/ or /home-b/ depending on the value of a cookie then we should be good to go. So let’s try that!

Using the following patch we can swap between /home-a/ and /home-b/ depending on the cookie value:

diff --git a/node_modules/gatsby/cache-dir/find-path.js b/node_modules/gatsby/cache-dir/find-path.js
index a231735..5dd8f90 100644
--- a/node_modules/gatsby/cache-dir/find-path.js
+++ b/node_modules/gatsby/cache-dir/find-path.js
@@ -109,6 +109,35 @@ export const grabMatchParams = rawPathname => {
   return {}
 }
 
+function getCookies() {
+  var c = document.cookie, v = 0, cookies = {};
+  if (document.cookie.match(/^\s*\$Version=(?:"1"|1);\s*(.*)/)) {
+      c = RegExp.$1;
+      v = 1;
+  }
+  if (v === 0) {
+      c.split(/[,;]/).map(function(cookie) {
+          var parts = cookie.split(/=/, 2),
+              name = decodeURIComponent(parts[0].trimLeft()),
+              value = parts.length > 1 ? decodeURIComponent(parts[1].trimRight()) : null;
+          cookies[name] = value;
+      });
+  } else {
+      c.match(/(?:^|\s+)([!#$%&'*+\-.0-9A-Z^`a-z|~]+)=([!#$%&'*+\-.0-9A-Z^`a-z|~]*|"(?:[\x20-\x7E\x80\xFF]|\\[\x00-\x7F])*")(?=\s*[,;]|$)/g).map(function($0, $1) {
+          var name = $0,
+              value = $1.charAt(0) === '"'
+                        ? $1.substr(1, -1).replace(/\\(.)/g, "$1")
+                        : $1;
+          cookies[name] = value;
+      });
+  }
+  return cookies;
+}
+
+function getCookie(name) {
+  return getCookies()[name];
+}
+
 // Given a raw URL path, returns the cleaned version of it (trim off
 // `#` and query params), or if it matches an entry in
 // `match-paths.json`, its matched path is returned
@@ -134,6 +163,31 @@ export const findPath = rawPathname => {
     foundPath = cleanPath(rawPathname)
   }
 
+  // This matches the initial call to `findPath` where
+  // `rawPathname === /` and we need to determine the
+  // rewrite based on the user's cookie value.
+  if (rawPathname === '/') {
+    const cv = getCookie('test_bucket')
+    if (cv === 'a') {
+      foundPath = '/home-a'
+    }
+    if (cv === 'b') {
+      foundPath = '/home-b'
+    }
+  }
+
+  // Once we match interally we get another call to `findPath`
+  // where `rawPathname === /home-a` or `rawPathname === /home-b`
+  // depending on the cookie value. This makes sure to catch that
+  // case as well.
+  if (rawPathname === '/home-a') {
+    foundPath = '/home-a'
+  }
+
+  if (rawPathname === '/home-b') {
+    foundPath = '/home-b'
+  }
+
   pathCache.set(trimmedPathname, foundPath)
 
   return foundPath

Now when we load / Gatsby runs findPath with rawPathname === '/' and gets /home-a or /home-b depending on the value of the cookie. It will then navigate to one of those pages internally (without modifying the URL) and load the related page data and template. Then it will call findPath one more time with rawPathname === '/home-a' or rawPathname === 'home-b' and we need to make sure it stays on those pages, so we have a bit more code there to handle that case. So let’s test it!

With a b cookie value:

With an a cookie value:

We’re in business! It’s now showing the correct page at the correct URL. 🚀

You can use the patch-package module or something similar to apply this patch to your Gatsby dependency and you should be good to go!

There’s some final steps needed to pull this all together but they’re relatively simple. First off, we need to tell Netlify to put this Edge Function in front of the / path. We can do this in our netlify.toml file using the following:

# netlify.toml

[[edge_functions]]
  path = "/"
  function = "abtest"

And finally we need to make sure that trailingSlash: "always" is set in our gatsby-config.{js,ts} to make sure that all pages have a trailing slash. This is only necessary because we are rewriting to /home-a/ and /home-b/ . If you instead have already configured trailingSlash: "never" then you can just drop the trailing slash in your rewrites and you should be good to go.

// gatsby-config.ts

import type { GatsbyConfig } from "gatsby"

const config: GatsbyConfig = {
  // ...
  trailingSlash: "always",
  // ...
}

export default config

That’s it! We’re off to the races and ready to start A/B testing our changes. You’ll want to track the performance of your variants using something like Google Analytics’ custom dimensions but there’s a million guides out there on that so I’ll leave that part for another article. Remember to check the demo repo if you’re stuck or if you think this article is missing something.

EDIT: after consulting with the Gatsby team we were able to devise a way to do this without a patch to Gatsby core. While it does require rewriting the page-data.json files for your variants I would say avoiding patching core is the better, less intrusive option. We have not fully experimented with the caching implications of rewriting the page data but theoretically this approach should work. The new approach is now the HEAD commit on the GitHub demo repo.