Configuring Varnish for Magento

Magento is one of the most popular e-commerce platforms out there and has both a Community Edition and an Enterprise Edition.

Magento is very flexible and versatile and that comes at a cost: the performance of a Magento store with lots of products and lots of visitors is often quite poor without a decent caching layer.

Luckily Magento has built-in caching mechanisms and the full page cache system has native support for Varnish.

This tutorial is a step-by-step guide on how to configure Varnish for Magento.

This tutorial features Magento 2.4 Community Edition and Varnish 9.

1. Install and configure Varnish

If you already have your Magento store up-and-running, and you’re looking to use Varnish as a full-page cache, you’ll have to decide where to install Varnish:

You can install Varnish on a dedicated machine and point your DNS records to that server.
You can install Varnish on the same server as your Magento store.

In this tutorial we’re installing Varnish on the same server where the Magento store is located.

For a detailed step-by-step Varnish installation guide, we’d like to refer you to the install guide.

2. Reconfigure the web server port

The web server that is hosting your Magento store is most likely set up to handle incoming HTTP requests on port 80 and HTTPS requests on port 443. For Varnish caching to properly work, Varnish needs to listen on ports 80 and 443. This also means that your web server needs to be configured on another listening port. We’ll use port 8080 as the new web server listening port for plain HTTP and port 8443 for HTTPS.

Depending on the type of web server you’re using, different configuration files need to be modified. The following tutorial explains how to change the web server listening ports.

3. Enable Varnish Full-Page Cache in the Magento admin panel

To ensure that Magento is aware of Varnish, you need to enable Varnish as a caching application in Magento’s Full Page Cache configuration.

Here’s how to configure this:

Go to the Magento admin panel
Select Stores > Configuration > Advanced > System
Expand the Full Page Cache section
Select Varnish Cache as the Caching Application
Optionally modify the TTL for public content value to override the standard Time-To-Live
Expand the Varnish Configuration section
Optionally modify the Access list value to override the list of allowed hosts to purge the cache
Optionally modify the Backend host value to override the hostname of your Varnish backend
Optionally modify the Backend port value to override the port number of your Varnish backend
Optionally modify the Grace period value to override the amount of grace time Varnish applies
To confirm the changes, press the Save Config button
Click Export VCL for Varnish 6 to download to custom-generated VCL file.

In this tutorial we’re assuming that Varnish and Magento are hosted on the same server. This means the following default values can be used:

The Access list value can be set to localhost
The Backend host value can be set to localhost
The Backend port value can be set to 8080

Other values, such as TTL for public content and Grace period, can use their respective default values: 86400 for the TTL and 300 for the Grace period.

4. Deploy the custom Magento VCL

In the previous step we downloaded a custom-generated VCL. Please ensure that this file is deployed to /etc/varnish/default.vcl on your Varnish server.

An alternative solution is to run the following command:

sudo bin/magento varnish:vcl:generate --export-version=6 | sudo tee /etc/varnish/default.vcl > /dev/null

This command will regenerate the VCL file using Magento’s command line interface and export it using the Varnish 6 syntax. The output will be written to /etc/varnish/default.vcl, which is the standard VCL file.

Fixing the backend health checks for Magento 2.4

If you’re using Magento 2.4, your web server’s root folder will be configured to the location of your Magento pub folder.

When your Magento root folder would be /var/www/html, the web server’s root will be /var/www/html/pub. In older versions of Magento that was not the case and /var/www/html would have been the root folder.

The problem is that Magento’s generated VCL file still points to the /pub/health_check.php endpoint, as you can see in the VCL snippet below:

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

The probe that performs the health checks will return HTTP 404 errors and as a consequence Varnish will return HTTP 503 Backend fetch failed errors because it considers the backend to be unhealthy.

To fix this, change the .url property for the probe from /pub/health_check.php to /health_check.php as illustrated below:

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

You can also run the following command to perform a find and replace for you:

sudo sed -i "s/\/pub\/health_check.php/\/health_check.php/g" /etc/varnish/default.vcl

If you’re using an older version of Magento 2 that has the web root outside of the pub folder, you can ignore this fix.

Optimized Magento VCL file

Although the generated VCL file is pretty decent once the health check is fixed, there are still some optimizations that can be made.

This VCL file uses the xkey VMOD that is not packaged in older versions of Varnish. As of Varnish 9 XKey is fully packaged, allowing you to perform tag-based cache invalidation.

If you’re using an older version of Varnish, you can either download the source compile XKey yourself, our you can swap out XKey and use bans instead.

Here’s the optimized VCL file we recommend:

# Optimized VCL for Magento 2 with Xkey support

vcl 4.1;

import cookie;
import std;
import xkey;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "localhost";
    "127.0.0.1";
    "::1";
}

sub vcl_recv {
    # Remove empty query string parameters
    # e.g.: www.example.com/index.html?
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header if set
    if (req.http.Host ~ ":[0-9]+$") {
        set req.http.Host = regsub(req.http.Host, ":[0-9]+$", "");
    }

    # Sorts query string parameters alphabetically for cache normalization purposes,
    # only when there are multiple parameters
    if (req.url ~ "\?.+&.+") {
        set req.url = std.querysort(req.url);
    }

    # Reduce grace to the configured setting if the backend is healthy
    # In case of an unhealthy backend, the original grace is used
    if (std.healthy(req.backend_hint)) {
        set req.grace = 1h;
    }

    # Allow cache purge via Ctrl-Shift-R or Cmd-Shift-R for IP's in purge ACL list
    if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") {
        if (client.ip ~ purge) {
            set req.hash_always_miss = true;
        }
    }

    # Purge logic to remove objects from the cache
    # Tailored to Magento's cache invalidation mechanism
    # The X-Magento-Tags-Pattern value is matched to the tags in the X-Magento-Tags header
    # If X-Magento-Tags-Pattern is not set, a URL-based purge is executed
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405));
        }

        # If the X-Magento-Tags-Pattern header is not set, just use regular URL-based purge
        if (!req.http.X-Magento-Tags-Pattern) {
            return (purge);
        }

        # Full Page Cache flush
        if (req.http.X-Magento-Tags-Pattern == ".*") {
            # If Magento wants to flush everything with .* regexp, it's faster to remove them
            # using the 'all' tag. This tag is automatically added by this VCL when a backend
            # is generated (see vcl_backend_response).
            if (req.http.X-Magento-Purge-Soft) {
                set req.http.n-gone = xkey.softpurge("all");
            } else {
                set req.http.n-gone = xkey.purge("all");
            }
            return (synth(200, req.http.n-gone));
        } elseif (req.http.X-Magento-Tags-Pattern) {
            # replace "((^|,)cat_c(,|$))|((^|,)cat_p(,|$))" to be "cat_c,cat_p"
            set req.http.X-Magento-Tags-Pattern = regsuball(req.http.X-Magento-Tags-Pattern, "[^a-zA-Z0-9_-]+" ,",");
            set req.http.X-Magento-Tags-Pattern = regsuball(req.http.X-Magento-Tags-Pattern, "(^,*)|(,*$)" ,"");
            if (req.http.X-Magento-Purge-Soft) {
                set req.http.n-gone = xkey.softpurge(req.http.X-Magento-Tags-Pattern);
            } else {
                set req.http.n-gone = xkey.purge(req.http.X-Magento-Tags-Pattern);
            }
            return (synth(200, req.http.n-gone));
        }
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url == "/health_check.php") {
        return (pass);
    }


    # Media files caching
    if (req.url ~ "^/(pub/)?media/") {
            unset req.http.Https;
            unset req.http.X-Forwarded-Proto;
            unset req.http.Cookie;
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?static/") {
            unset req.http.Https;
            unset req.http.X-Forwarded-Proto;
            unset req.http.Cookie;
    }

    # Collapse multiple cookie headers into one.
    # We do this, because clients often send a Cookie header for each cookie they have.
    # We want to join them all together with the ';' separator, so we can parse them in one batch.
    std.collect(req.http.Cookie, ";");

    # Parse the cookie header.
    # This means that we can use the cookie functions to check for cookie existence,
    # values, etc down the line.
    cookie.parse(req.http.Cookie);

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(_branch_match_id|_bta_[a-z]+|_bta_c|_bta_tid|_ga|_gl|_ke|_kx|campid|cof|customid|cx|dclid|dm_i|ef_id|epik|fbclid|gad_source|gbraid|gclid|gclsrc|gdffi|gdfms|gdftrk|hsa_acc|hsa_ad|hsa_cam|hsa_grp|hsa_kw|hsa_mt|hsa_net|hsa_src|hsa_tgt|hsa_ver|ie|igshid|irclickid|matomo_campaign|matomo_cid|matomo_content|matomo_group|matomo_keyword|matomo_medium|matomo_placement|matomo_source|mc_[a-z]+|mc_cid|mc_eid|mkcid|mkevt|mkrid|mkwid|msclkid|mtm_campaign|mtm_cid|mtm_content|mtm_group|mtm_keyword|mtm_medium|mtm_placement|mtm_source|nb_klid|ndclid|origin|pcrid|piwik_campaign|piwik_keyword|piwik_kwd|pk_campaign|pk_keyword|pk_kwd|redirect_log_mongo_id|redirect_mongo_id|rtid|s_kwcid|sb_referer_host|sccid|si|siteurl|sms_click|sms_source|sms_uph|srsltid|toolid|trk_contact|trk_module|trk_msg|trk_sid|ttclid|twclid|utm_[a-z]+|utm_campaign|utm_content|utm_creative_format|utm_id|utm_marketing_tactic|utm_medium|utm_source|utm_source_platform|utm_term|vmcid|wbraid|yclid|zanpid)=") {
        set req.url = regsuball(req.url, "(_branch_match_id|_bta_[a-z]+|_bta_c|_bta_tid|_ga|_gl|_ke|_kx|campid|cof|customid|cx|dclid|dm_i|ef_id|epik|fbclid|gad_source|gbraid|gclid|gclsrc|gdffi|gdfms|gdftrk|hsa_acc|hsa_ad|hsa_cam|hsa_grp|hsa_kw|hsa_mt|hsa_net|hsa_src|hsa_tgt|hsa_ver|ie|igshid|irclickid|matomo_campaign|matomo_cid|matomo_content|matomo_group|matomo_keyword|matomo_medium|matomo_placement|matomo_source|mc_[a-z]+|mc_cid|mc_eid|mkcid|mkevt|mkrid|mkwid|msclkid|mtm_campaign|mtm_cid|mtm_content|mtm_group|mtm_keyword|mtm_medium|mtm_placement|mtm_source|nb_klid|ndclid|origin|pcrid|piwik_campaign|piwik_keyword|piwik_kwd|pk_campaign|pk_keyword|pk_kwd|redirect_log_mongo_id|redirect_mongo_id|rtid|s_kwcid|sb_referer_host|sccid|si|siteurl|sms_click|sms_source|sms_uph|srsltid|toolid|trk_contact|trk_module|trk_msg|trk_sid|ttclid|twclid|utm_[a-z]+|utm_campaign|utm_content|utm_creative_format|utm_id|utm_marketing_tactic|utm_medium|utm_source|utm_source_platform|utm_term|vmcid|wbraid|yclid|zanpid)=[-_A-z0-9+(){}%.*]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Bypass authenticated GraphQL requests without a X-Magento-Cache-Id
    if (req.url ~ "/graphql" && !req.http.X-Magento-Cache-Id && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    # For non-graphql requests we add the value of the Magento Vary cookie to the
    # object hash. This vary cookie can contain things like currency, store code, etc.
    # These variations are typically rendered server-side, so we need to cache them separately.
    if (req.url !~ "/graphql" && cookie.isset("X-Magento-Vary")) {
        hash_data(cookie.get("X-Magento-Vary"));
    }

    # To make sure http users don't see ssl warning
    hash_data(req.http.X-Forwarded-Proto);

    # For graphql requests we execute the process_graphql_headers subroutine
    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }

    # Don't strip trailing slash for the homepage
    if(req.url == "/") {
        hash_data(req.url);
    } else {
        # Strip trailing slash from URL for cache normalization
        hash_data(regsub(req.url, "\/$", ""));
    }

    # Always include the host header in the hash to separate different websites
    hash_data(req.http.Host);

    return(lookup);
}

sub process_graphql_headers {
    # The X-Magento-Cache-Id header is used by graphql clients to let the backend
    # know which variant it is. It's basically the same as the Vary # cookie, but
    # for graphql requests.
    if (req.http.X-Magento-Cache-Id) {
        hash_data(req.http.X-Magento-Cache-Id);

        # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
        if (req.http.Authorization ~ "^Bearer") {
            hash_data("Authorized");
        }
    }

    # If store header is specified by client, add it to the hash
    if (req.http.Store) {
        hash_data(req.http.Store);
    }

    # If content-currency header is specified, add it to the hash
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {
    # Serve stale content for one day after object expiration while a fresh
    # version is fetched in the background.
    set beresp.grace = 1d;

    if (beresp.http.X-Magento-Tags) {
        # set comma separated xkey with "all" tag, allowing for fast full purges
        set beresp.http.XKey = beresp.http.X-Magento-Tags + ",all";
        unset beresp.http.X-Magento-Tags;
    }

    # All text-based content can be parsed as ESI
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    # Cache HTTP 200 responses
    if (beresp.status != 200) {
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
        return (deliver);
    }

    # Don't cache if the request cache ID doesn't match the response cache ID for graphql requests
    if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
       set beresp.ttl = 120s;
       set beresp.uncacheable = true;
       return (deliver);
    }

    # Remove the Set-Cookie header for cacheable content
    # Only for HTTP GET & HTTP HEAD requests
    # We remove the Set-Cookie header from the VCL response, because we want to keep
    # the objects in the cache anonymous.
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.Set-Cookie;
    }
}

sub vcl_deliver {
    if (obj.uncacheable) {
        set resp.http.X-Magento-Cache-Debug = "UNCACHEABLE";
    } else if (obj.hits > 0 && obj.ttl > 0s) {
        set resp.http.X-Magento-Cache-Debug = "HIT";
    } else if (obj.hits > 0 && obj.ttl <= 0s) {
        set resp.http.X-Magento-Cache-Debug = "HIT-GRACE";
    } else if(req.hash_always_miss) {
        set resp.http.X-Magento-Cache-Debug = "MISS-FORCED";
    } else {
        set resp.http.X-Magento-Cache-Debug = "MISS";
    }

    # Let browser and Cloudflare cache non-static content that are cacheable for short period of time
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(media|static)/" && obj.ttl > 0s && !obj.uncacheable) {
        set resp.http.Cache-Control = "must-revalidate, max-age=60";
        set resp.http.Cache-Control = "no-store, must-revalidate, max-age=60";
    }

    # Remove all headers that don't provide any value for the client
    unset resp.http.XKey;
    unset resp.http.Expires;
    unset resp.http.Pragma;
    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_synth {
    if(req.method == "PURGE")  {
        set resp.http.Content-Type = "application/json";
        if(req.http.X-Magento-Tags-Pattern) {
            set resp.body = {"{ "invalidated": "} + resp.reason + {" }"};
        } else {
            set resp.body = {"{ "invalidated": 1 }"};
        }
        set resp.reason = "OK";
        return(deliver);
    }
}

Swapping out xkey for bans on older versions of Varnish

If you’re running an older version of Varnish that doesn’t package the xkey VMOD, you can swap out the XKey-based cache invalidation logic and replace it with the more traditional ban logic.

Just remove the following lines of code from the VCL template:

# Full Page Cache flush
if (req.http.X-Magento-Tags-Pattern == ".*") {
    # If Magento wants to flush everything with .* regexp, it's faster to remove them
    # using the 'all' tag. This tag is automatically added by this VCL when a backend
    # is generated (see vcl_backend_response).
    if (req.http.X-Magento-Purge-Soft) {
        set req.http.n-gone = xkey.softpurge("all");
    } else {
        set req.http.n-gone = xkey.purge("all");
    }
    return (synth(200, req.http.n-gone));
} elseif (req.http.X-Magento-Tags-Pattern) {
    # replace "((^|,)cat_c(,|$))|((^|,)cat_p(,|$))" to be "cat_c,cat_p"
    set req.http.X-Magento-Tags-Pattern = regsuball(req.http.X-Magento-Tags-Pattern, "[^a-zA-Z0-9_-]+" ,",");
    set req.http.X-Magento-Tags-Pattern = regsuball(req.http.X-Magento-Tags-Pattern, "(^,*)|(,*$)" ,"");
    if (req.http.X-Magento-Purge-Soft) {
        set req.http.n-gone = xkey.softpurge(req.http.X-Magento-Tags-Pattern);
    } else {
        set req.http.n-gone = xkey.purge(req.http.X-Magento-Tags-Pattern);
    }
    return (synth(200, req.http.n-gone));
}

And replace it with the following code:

ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
return (synth(200, "0"));

While the code is a lot simpler, we do prefer XKey to perform tag-based cache invalidation: bans don’t perform that well at scale, and XKey supports soft purging.

Soft purging is a big one in Magento: the impact of cache purges for categories with lots of associated products is pretty big. Soft purging will mark the object as expired, instead of directly removing it from the cache. This means that the next visitor gets a (stale) cache hit, while Varnish asynchronously fetches the updated version of the content from Magento.

If you must, you can use the ban() function instead of XKey. Our advice is to upgrade to Varnish 9 or a more recent version.

Use the magento2-varnish-extended module for Magento

You can use the optimized VCL file for Magento that we provide in this tutorial.

It also makes sense to use the magento2-varnish-extended module for Magento. This module is developed by Elgentos and the broader Magento community. Varnish Software contributed to this effort by writing the VCL file and the VTC tests under the supervision of Magento community leaders.

This module is highly configurable and dynamically parses values into the VCL file.

If our optimized VCL file for Magento has specific features you don’t need, magento2-varnish-extended may give you the option to disable them in the Magento admin panel.

5. Restart the services

If you’re using Apache as a web server, you’ll run the following command to restart it:

sudo systemctl restart apache2

If you’re using Nginx instead, please run the following command to restart your web server:

sudo systemctl restart nginx

And finally, you’ll have to run the following command to restart Varnish:

sudo systemctl restart varnish

After the restart, your web server will accept traffic on port 8080, Varnish will handle HTTP traffic on port 80. The restart will also ensure the right VCL file is loaded, which will ensure that requests for your Magento store can be properly cached.

6. Making cache purges work

By default Magento will send HTTP PURGE requests to Varnish using the base url. This is not done through the loopback interface but through one of the main network interfaces. The IP address the purge requests originate from doesn’t match localhost and will cause the purge to fail.

Assuming Varnish is installed on the same machine as Magento, listening on port 80, the following command can be executed to make Magento aware of Varnish:

bin/magento setup:config:set --http-cache-hosts=localhost

This will ensure the loopback interface is used and the incoming requests pass the ACL, which is configured to only allow connections coming from localhost.

If Varnish is located on another server or another port, you have to set the --http-cache-hosts accordingly. Click here for more information.

7. Flushing the cache

In the final step we will flush all Magento caches to ensure a consistent state:

bin/magento cache:flush

Configuring Varnish for Magento

1. Install and configure Varnish#

2. Reconfigure the web server port#

3. Enable Varnish Full-Page Cache in the Magento admin panel#

4. Deploy the custom Magento VCL#

Fixing the backend health checks for Magento 2.4#

Optimized Magento VCL file#

Swapping out xkey for bans on older versions of Varnish#

Use the magento2-varnish-extended module for Magento#

5. Restart the services#

6. Making cache purges work#

7. Flushing the cache#

1. Install and configure Varnish

2. Reconfigure the web server port

3. Enable Varnish Full-Page Cache in the Magento admin panel

4. Deploy the custom Magento VCL

Fixing the backend health checks for Magento 2.4

Optimized Magento VCL file

Swapping out xkey for bans on older versions of Varnish

Use the magento2-varnish-extended module for Magento

5. Restart the services

6. Making cache purges work

7. Flushing the cache