Changes in Varnish-Cache 7.7

For information about updating your current Varnish deployment to the new version, see Upgrading to Varnish-Cache 7.7.

NOTE: In this Varnish-Cache release, we changed how timestamps are taken for the http2 protocol, which could look like a performance regression, but is not. See http2 related timestamps.

A more detailed and technical account of changes in Varnish-Cache, with links to issues that have been fixed and pull requests that have been merged, may be found in the change log.

Security

VSV 15:

The client connection is now always closed when a malformed request is received.

varnishd

Parameters

The bitfield parameters debug, experimental, feature, vcc_feature and vsl_mask now consistently support the special values all and none. The output format of varnishadm param.show has been adjusted to always output the parameter value relative to either all or none in a format which would also be accepted by varnishadm param.set and the varnishd -p option.

The new http_req_overflow_status parameter now allows to optionally send a response with a status between 400 and 499 (inclusive) if a request exceeds http_req_size. The default of 0 keeps the existing behavior to just close the connection in this case.

The new ban_any_variant parameter allows to configure the maximum number of possibly non matching variants evaluated against the ban list during lookup. The default value of 10000 avoids excessive time spent for ban checks during lookups, which could cause noticeable delays for cases with a very high number of bans and/or variants (in the 1000s).

Setting ban_any_variant to 0 changes the behavior of the lookup-time ban check to only consider matching objects for tests against the ban list, which can be considered a bugfix, depending on the exact interpretation of the semantics of ban expressions with regards to variants. 0 will become the new default in a future release of Varnish-Cache.

The varnishd parameters max_restarts and max_retries have been made more consistent, which prevents a potential panic that could be triggered when reducing the value of max_restarts under certain conditions.

Jails

The linux jail gained control of transparent huge pages (THP) settings: The transparent_hugepage suboption can be set to ignore to do nothing, enable to enable THP (actually, disable the disable), disable to disable THP or try-disable to try do disable, but not emit an error if disabling fails. try-disable is the default.

Error handling from the jail subsystem has been streamlined to avoid some confusing and/or contradictory error messages as well as turn assertion failures into error messages.

Other changes in varnishd

An issue has been fixed which could cause a crash when varnishd receives an invalid Content-Range header from a backend.

The management process now logs the PID of the process from which it received a signal.

Changes to VCL

VCL variables

Two new VCL variables req.filters and bereq.filters can now be used to pass request bodies through a list of VFPs and backend request bodies through a list of VDPs respectively. This can be useful for processing or transforming request bodies as they go through varnish. See VCL-Variables for more details.

VCL now supports unset req.grace and unset req.ttl to reset the respective variables to the “no effect” value, which is also the default.

The scope of VCL variables req.is_hitmiss and req.is_hitpass is now restricted to vcl_miss, vcl_deliver, vcl_pass, vcl_synth and vcl_pass, vcl_deliver, vcl_synth, respectively.

The Content-Length header is now consistently removed after unset bereq.body on the backend side.

req.hash is now also readable from vcl_synth and vcl_pipe.

Other changes to VCL

Behavior of the VCL include statement with the +glob option has been clarified to not search directories in vcl_path. Using +glob includes with a relative path that does not start with “./” will now result in a VCL compile failure.

Validation of the PROXY2 PP2_TYPE_AUTHORITY TLV sent with .via backends has been corrected: IP addresses are no longer accepted as an authority and port numbers are automatically removed.

return (fail(...)) can now take strings returned from a vmod.

varnishadm

  • The CLI command backend.list -j now outputs IPs/port information.

Generic Logging (VSL)

affecting varnishlog, varnishncsa and varnishtop:

Timestamps for http/2 requests have been corrected and made similar to how they are taken for http/1.

For http/1, the start time, internally called “t_first”, is taken as soon as any part of the request (headers) is received. Previously, http/2 took it later, possibly much later if long header lines were involved. http/2 now takes it the same way as http/1 when the first bit of the first HEADERS frame of the request arrives.

Timing behavior for http/1 and http/2 is different and can not be directly compared. But with this change, the Timestamp VSL records for http/2 now at least reflect reality better.

NOTE that after upgrading Varnish-Cache, processing and response times for http/2 will now be reported as worse than before the upgrade, potentially much worse. This is NOT a performance regression, but rather due to the corrected timestamps, which arguably were wrong for http/2.

http2 logging

For http/2, normal client behavior like timeouts or closed connection was logged with a SessError tag and ENHANCE_YOUR_CALM in additional Debug log records. This behavior was misleading and has been corrected.

http/2 error detail reporting in Debug log records has been clarified: Connection errors are now prefixed with H2CE_, and stream errors with H2SE_, respectively.

http/2 BogoHeader log records now contain the first offending byte value in hex.

Interactive mode in varnishstat, varnishtop and varnishhist

Handling of curses errors in the interactive mode of varnishstat, varnishtop and varnishhist has been streamlined and one wrong assertion has been fixed, which could cause a crash with certain terminal types as set through the TERM environment variable.

varnishncsa

The hitmiss and hitpass handling indicators have been added to the Varnish:handling format of varnishncsa.

varnishncsa now handles headers unset and changed from VCL more consistently: request headers are logged as they were received from the client and as they were sent to the backend, while response headers are logged as they were sent to the client and as they were received from the backend.

varnishstat

Pressing the 0 key in varnishstat interactive (curses) mode now resets averages.

The backend happy VSC bitfield is now set to all ones for backends with no configured probe.

varnishtest

varnishtest can now send arbitrary http/2 settings frames and arbitrary PROXY2 tlvs.

varnishtest has been changed to always set a VARNISH_DEFAULT_N environment variable to ensure that varnish invoked from varnishtest always has a valid workdir.

Changes for developers and VMOD authors

miniobj.h: Helper macros SIZEOF_FLEX_OBJ() and ALLOC_FLEX_OBJ() have been added to facilitate use of structs with flexible array members.

The acceptor code has been refactored for basic support of pluggable acceptors.

Two fields have been added to the VMOD data registered with varnish-cache:

  • vcs for Version Control System is intended as an identifier from the source code management system, e.g. the git revision, to identify the exact source code which was used to build a VMOD binary.
  • version is intended as a more user friendly identifier as to which version of a vmod a binary represents.

The panic output and the debug.vmod CLI command output now contain these identifiers.

Where supported by the compiler and linker, the vcs identifier is also reachable via the .vmod_vcs section of the vmod shared object ELF file and can be extracted, for example, using readelf -p.vmod_vcs <file>

To set the version, vmodtool.py now accepts a $Version stanza in vmod vcc files. If $Version is not present, an attempt is made to extract PACKAGE_STRING from an automake Makefile, otherwise NOVERSION is used as the version identifier.

A new facility has been added allowing transport delivery functions to disembark the worker thread which had been handling a request’s VCL code during delivery by returning VTR_D_DISEMBARK from the vtr_deliver_f function.

This will enable future optimizations to make transport protocol code more efficient.

To enable this facility, a new request processing step finish has been added once delivery is complete.

eof