Introduction: how does Varnish work?

Varnish is considered a reverse caching HTTP proxy: it’s a proxy server that sits in front of the web servers, caches their content, and serves the cached content to the requesting clients

┌─────────┐
│ browser │
└─────────┘                                 ┌─────────┐
           \                               ┌─────────┐│
┌─────┐     ╔═════════╗    ┌─────────┐    ┌─────────┐│┘
│ app │ --- ║ Network ║ -- | Varnish │ -- │ Backend │┘
└─────┘     ╚═════════╝    └─────────┘    └─────────┘
            /
┌────────────┐
│ API-client │
└────────────┘

However, Varnish does more than just caching: it provides policy, analytics, visibility and mitigation for your web traffic.

It can serve as a load balancer, an HTTP proxy, an observability gateway. It can even be considered an edge computing platform thanks to its programmability.

VCL

For each and every request, Varnish runs through the VCL program to decide what should happen.

VCL is short for Varnish Configuration Language. It is the domain-specific programming language that Varnish uses to define the behavior of the cache. It is not a top-down programming language that requires all behavior to be defined, but a language that hooks into the Finite State Machine of Varnish.

The built-in VCL defines the default behavior of Varnish, and it can be extended/overridden by writing your own VCL.

Defining backends

Varnish proxies requests to one or more backends and caches backend responses. In most cases, you’ll define a backend that Varnish uses to proxy requests to.

Here’s a simple example of such a backend definition:

vcl 4.1;

backend default {
    .host = "www.varnish.org";
    .port = "80";
}

The dedicated backend tutorial explains backends in greater detail. There is also reference documentation on backend definitions.

You can define multiple backends, and define policies on when to use what backend:

sub vcl_recv {
   if (req.url ~ "^/images/") {
       set req.backend_hint = image_server;
   } else {
       set req.backend_hint = application_server;
   }
}

In this case all requests with a URL that start with /images/ is routed to the image_server backend. All other requests end up on the application_server backend.

You can do a lot more with backends, you can even loadbalance between them. Have a look at our tutorial about using multiple backends to learn more.

Setting cache policies

Varnish supports HTTP caching best practices and understands the Cache-Control HTTP header if your backend server sends one.

If Varnish sees the following response header, it will cache the response for an hour:

Cache-Control: public, max-age=3600

However, backends don’t always follow the standards. So ultimately the VCL program makes the decision to cache and how long, and if you want to send a different Cache-Control header to the clients, VCL can do that too.

You can set the lifetime of cached objects in the vcl_backend_response subroutine and you can make it context-specific:

sub vcl_backend_response {
    if(beresp.http.content-type ~ "^image/") {
        set beresp.ttl = 1y;
    } elseif(beresp.http.cache-control !~ "(s-)?max-?age=[0-9]+") {
        set beresp.ttl = 1h;
    }
 }

This example caches images for a year. The code qualifies a response as an image if the Content-Type response header starts with image/. Other responses will be cached for an hour, unless the response contains a Cache-control header that has a max-age or s-maxage value.

Definining cache policies in VCL is more than just setting the cache lifetime of an object. The built-in VCL decides what can be cached and what not, and in most real-world use cases, you have to override this behavior and define your own policies.

Here’s an example:

sub vcl_recv {
    if(req.url ~ "^/admin($|/.*)") {
        return(pass);
    }
    unset req.http.Cookie;
}

This VCL snippet will bypass the cache for the /admin pages, and will remove cookies for all other pages. There is also a dedicated tutorial on URL-based cache bypassing.

VCL compilation

When you load the VCL program into Varnish, it is compiled into a C-program which is compiled into a shared library, which varnish then loads and calls into, therefore VCL code is fast.

Unless configured otherwise, the /etc/varnish/default.vcl file is compiled when Varnish starts.

You can load multiple VCL files and dynamically load VCL programs without restarting Varnish. The Varnish CLI offers commands like vcl.list, vcl.load, vcl.use, and vcl.discard to manage VCL programs that you can run with varnishadm.

The varnishreload script, that is deployed with Varnish, wraps around the CLI commands and offers an easy way to reload VCL.

Invalidating the cache

Caching for too long is far more dangerous than not caching enough. When the data on the origin server changes, the cached content is stale and in need of revalidation. Meanwhile end-users only see the old content, which can cause issues.

Setting a low TTL value is not the real solution. Luckily, Varnish comes with extensive caching invalidation functionality:

Varnish offers purging capabilities to remove individual objects from the cache
Varnish has a ban() function to remove multiple objects from the cache through expressions
Varnish offers tag-based invalidation functionality to mark objects with arbitrary keys and remove objects from the cache based on these keys

Here’s a simple example where the return(purge) is used to remove a single object from the cache:

vcl 4.1;

acl purge {
    "localhost";
    "192.168.55.0"/24;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method Not Allowed"));
        }
        return (purge);
    }
}

This VCL snippet will capture requests that use the PURGE request method and purge the corresponding object from the cache.

The following HTTP request can be used to trigger a cache purge for the homepage:

PURGE / HTTP/1.1
Host: example.com

We recommend checking out our cache invalidation tutorial, that covers the topic in more detail.

Logging

Everything Varnish does is recorded in VSL log records, which can be examined and monitored in real time or recorded for later use in native or NCSA format, and when we say everything we mean everything:

*   << Request  >> 318737
-   Begin          req 318736 rxreq
-   Timestamp      Start: 1612787907.221931 0.000000 0.000000
-   Timestamp      Req: 1612787907.221931 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       192.0.2.24 39698 a1
-   ReqMethod      GET
-   ReqURL         /vmods/
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: varnish-cache.org
-   ReqHeader      Accept: text/html, application/rss+xml, […]
-   ReqHeader      Accept-Encoding: gzip,deflate
-   ReqHeader      Connection: close
-   ReqHeader      User-Agent: Mozilla/5.0 […]
-   ReqHeader      X-Forwarded-For: 192.0.2.24
-   VCL_call       RECV
-   VCL_acl        NO_MATCH bad_guys
-   VCL_return     hash
[…]

These VSL log records are written to a circular buffer in shared memory, from where other programs can subscribe to them via a supported API. One such program is varnishncsa which produces NCSA-style log records:

192.0.2.24 - - [08/Feb/2021:12:42:35 +0000] "GET http://vmods/ HTTP/1.1" 200 0 […]

Edge Side Includes

Varnish supports ESI - Edge Side Includes which makes it possible to send responses to clients which are composed of different bits from different backends, with the very important footnote that the different bits can have very different caching policies.

With ESI a backend can tell Varnish to edit the content of another object into a HTML page:

<h1>Todays Top News</h1>
<esi:include src="/topnews" />

The /topnews request will be handled like every other request in Varnish: VCL will decide if it can be cached, which backend should supply it and so on.

Even if the whole object in the example can not be cached, for instance if the page is dynamic content for a logged-in user, the /topnews object can be cached and can be shared from the cache, between all users.

Because VCL is in full control of every request, and because VCL can be changed instantly on the fly, Varnish is a great tool to implement both reactive and prescriptive content policies.

Varnish, and VCL is particular, are well suited to sort requests and collect metrics for real-time A/B testing or during migrations to a new backend system.

Reactive content-policies can be anything from blocking access to an infected backend or fixing the URL from the QR code on the new product, to extending caching times while the backend rebuilds the database.

Introduction: how does Varnish work?

VCL#

Defining backends#

Setting cache policies#

VCL compilation#

Invalidating the cache#

Logging#

Edge Side Includes#

VCL