Vinyl Cache recipe for buildout

plone.recipe.vinylcache is a zc.buildout recipe to install Vinyl Cache (formerly known as Varnish Cache; the upstream project renamed itself in early 2026). Even though the name contains the name Plone, there is nothing Plone-specific about this recipe: it works for non-Zope sites just as well.

This recipe is a fork of plone.recipe.varnish, adapted for the new project name and the current Vinyl Cache 9.0.x stable release line. It supports Vinyl Cache 9.0.x only. If you are still running the previous Varnish 6.0 LTS line, keep using plone.recipe.varnish instead -- that package is unaffected by this fork and continues to target 6.0 LTS.

Note: as of the 9.0.x release line, the rename is a project/brand rename -- the daemon this recipe builds and runs is still the varnishd binary (the source tree's internal directory layout has been renamed, e.g. bin/vinyld/, but the compiled/installed program, its -V self-identification, its Server:/X-Varnish HTTP headers, and the varnish-modules VMOD collection are all still named/branded "varnish" as of 9.0.3). This recipe therefore still refers to the daemon as varnishd throughout; only the recipe's own package name and download defaults track the new "Vinyl Cache" branding and release line.

What's new compared to plone.recipe.varnish

Besides targeting Vinyl Cache 9.0.x instead of Varnish 6.0 LTS, this fork adds a handful of things not present in plone.recipe.varnish:

• The default vcl_hash now includes req.http.host (matching Vinyl Cache's own built-in default), avoiding cache cross-contamination between different vhosts/backends serving overlapping URL paths on the same instance.
• PATCH is treated like PUT/POST/DELETE in the method whitelist, and WebSocket upgrade requests are piped through instead of hitting normal GET/HEAD caching logic.
• Accept-Encoding is normalized to reduce cache fragmentation, and large files (by extension) are streamed (beresp.do_stream) instead of piped, keeping them cacheable and visible to logging.
• A new shard value for the balancer option (consistent-hashing director, better cache hit ratio than round_robin/random across several backends).
• The purge ACL suppresses Vinyl Cache 9.0's (harmless but noisy) ACL-folding compiler warning for common setups.
• verbose-headers is a real, working option (diagnostic X-Cache/X-Cacheable/grace response headers, off by default).
• A new purge-by-id option, compatible with collective.purgebyid, for purging every cached variant of a piece of content by id rather than needing to enumerate cached URLs -- via ban() (default, no vmod needed) or via the xkey vmod (opt-in, more efficient).
• A new max-cacheable-size option: objects larger than this (by Content-Length) are never cached, preventing a single large object from evicting many smaller ones under LRU pressure.
• A new tls-config option (script) mapping to varnishd -A, a Vinyl Cache 9.0 addition letting varnishd terminate TLS itself.
• A new plone.recipe.vinylcache:selfsigned recipe to generate a matching self-signed certificate for internal/dev/testing use of tls-config.

See CHANGES.rst for the full list, including bugfixes carried over from fixing dead/no-op options this fork was originally forked with.

Configuring it is very simple. For example:

[varnish-build]
recipe = plone.recipe.vinylcache:build

[varnish-configuration]
recipe = plone.recipe.vinylcache:configuration
backends = 127.0.0.1:8081

[varnish-script]
recipe = plone.recipe.vinylcache:script
bind = 127.0.0.1:8000
cache-size = 512M

This configures three buildout parts:

varnish-build

which will download, compile and install Vinyl Cache,

varnish-configuration

which generates the VCL configuration file, sending requests to a backend at 127.0.0.1:8081, and

varnish-script

which runs the varnishd daemon, configured to listen on 127.0.0.1:8000 for requests, using a 512 megabyte cache.

A wrapper script for the varnishd startup command is created in the bin directory of your buildout.

Please note that the configuration generated by this recipe is specifically meant for Vinyl Cache 9.0.x, and generates VCL syntax version 4.1.

Area's of interest to look at when you use this recipe

Even though this recipe tries to provide sane defaults for most parameters for a drop in placement, there are a few area's of expertise which you can learn about to understand and test for a performant but stable Vinyl Cache set up.

• If you use this recipe, you will use buildout and a common setup for 'single' projects is that your Vinyl Cache will only cache for one Plone site. An inherent problem with caching is cache invalidation: when an editor changes a page you want to make sure the next delivery for this content item fetched freshly from the backend (Plone) site. Plone's caching setup can be configured to send a purge requests for changed content to Vinyl Cache. The calculated has plays an essential role in this type of purging: if the calculated hash from the client request is different from the calculated hash on the purge request, purging will fail.
• There are clever alternative purge request setups, which can improve freshness but you really have to know what you are doing and experience so far is that more advanced schemes have broken between Varnish/Vinyl Cache upgrades.
• Especially if you have multiple backends and you let Vinyl Cache do the load balancing, don't forget to enable the grace-sick and grace-healthy options. When the recipe notices there parameters, it automatically enables health probe settings in the generated vcl. grace helps serving pages temporarily when your backends are temporarily down.
• As soon as a cookie appears on an incoming request, Vinyl Cache will not cache the request and even store the url in a 'hit for pass' buffer so that subsequent similar requests don't get delayed by waiting in the backend request queue. The generated vcl has a function which strips off most irrelevant cookies from incomiing requests before they get passed to the backend to increase cache hit rate. the __ac cookie is the most notable exception, this indicates for Plone that a user is logged in and caching should be disabled.
• You can monitor Vinyl Cache caching operations in great detail by learning how to use varnishlog and the query language, but it will take at least a few hours if you have never used this tool before. It is easy to draw the wrong conclusions from just poking around a bit in varnishlog and seeing hits or misses.

Virtual hosting

Vinyl Cache supports virtual hosting by selecting a different backend server based on headers on the incoming request. You can configure the backends through the backends option:

[varnish-configuration]
backends =
   plone.org:127.0.0.1:8000
   plone.net:127.0.0.1:9000

This will generate a configuration which sends all traffic for the plone.org host to a backend server running on port 8000 while all traffic for the plone.net host is send to port 9000.

Zope 2 hosting (with Virtual Host Monster)

If you are using Zope 2 as backend server you will need to rewrite the URL so the Zope Virtual Host Monster (VHM) can generate correct links for links in your pages. This can be done either by a web server such as Apache or nginx (placed either in front or behind Vinyl Cache) but can also be done by Vinyl Cache itself.

The three options are described below.

Option 1 (rewrites after Vinyl Cache)

If generating these VHM-style URLs in a proxy behind Vinyl Cache (or if using VHM's 'mapping' feature), no extra Vinyl Cache configuration is needed. Just make sure the backends option directs the traffic to the proxy.

Option 2 (rewrites before Vinyl Cache)

If generating these VHM-style URLs in a proxy in front of Vinyl Cache, no extra Vinyl Cache configuration is needed as long as the original hostname is still retained in the URL. If the hostname is not retained, you can tell Vinyl Cache to direct requests based on the "path" instead of the hostname. For example:

[varnish-configuration]
backends =
  /VirtualHostBase/http/plone.org:80/Plone:127.0.0.1:8000
  /VirtualHostBase/http/plone.net:80/Plone:127.0.0.1:9000

This will generate a configuration which sends all traffic for any request whose path starts with /VirtualHostBase/http/plone.org:80/Plone to a backend server running at 127.0.0.1 on port 8000, while request paths starting with /VirtualHostBase/http/plone.net:80/Plone are sent to port 9000.

Option 3 (rewrites within Vinyl Cache)

To have Vinyl Cache generate these VHM-style URLs, you can use the zope2_vhm_map option. Here is an example:

[varnish-configuration]
zope2_vhm_map =
    plone.org:/plone
    plone.net:/plone

This tells us that the domain plone.org should be mapped to the location /plone in the backend. By combining this with the information from the backends option a configuration will be generated that maps URLs correctly.

Load Balancing

Vinyl Cache supports load balancing by configuring a director for a pool of backends. This director sends the incoming requests that cannot be fulfilled by Vinyl Cache to backends in the pool in either random or round robin fashion. You can configure the director via the balancer option:

[varnish-configuration]
balancer = random

This will generate a configuration which sends all traffic to the director, which will choose a 'random' backend server to fulfill the request if the content requested is not cached by Vinyl Cache itself.

plone.recipe.vinylcache reference

The plone.recipe.vinylcache recipe does one or more of the following:

plone.recipe.vinylcache:build

compiles Vinyl Cache from sources

plone.recipe.vinylcache:configuration

generates a VCL-configuration file

plone.recipe.vinylcache:script

generates a wrapper script inside your buildout that will start Vinyl Cache with the correct configuration.

plone.recipe.vinylcache:selfsigned

generates a self-signed TLS certificate and a config file suitable for script's tls-config option.

Build Vinyl Cache from sources

build is based on zc.recipe.cmmi - so all parameters from that recipe are available here too (but rarely used). These options are available for the recipe part plone.recipe.vinylcache:build.

Three parameters are different/ extra:

url

Location used for download of Vinyl Cache sources. Defaults to the current stable 9.0.x release.

jobs

Passes the number of parallel jobs to make, defaults to 4. Adjust as needed to your CPU resources.

compile-vmods

Boolean flag defaults to False, used for building Varnish/Vinyl Cache modules. By defaults, the modules are from the module collection by Varnish Software (note: that module collection dropped support for Varnish 6.0 LTS; the release this recipe defaults to targets the 9.0.x line).

VCL Configuration Generator

These options are available for the recipe part plone.recipe.vinylcache:configuration.

backends

Specifies the backend or backends which will process the (uncached) requests. The syntax for backends:

[<hostname>][/<path>]:<ip address>:<port>

The optional hostname and path allows you to do virtual hosting. If multiple backends are specified then each backend must include either a hostname or path (or both) so that Vinyl Cache can direct the matching request to the appropriate backend. Defaults to 127.0.0.1:8080.

balancer

If included and set to random, round_robin or shard, this option configures Vinyl Cache to load balance the servers specified by the backends directive. Possible values: none (default), round_robin, random or shard. shard uses Vinyl Cache's consistent-hashing director: the same request (by default keyed on client.identity) always lands on the same backend, which gives a much better cache hit ratio than round_robin/ random when several backends could each independently cache the same content.

between-bytes-timeout

If specified, this option configures the timeout (in seconds) for Vinyl Cache waiting between bytes when receiving data from a backend. Vinyl Cache will only wait this many seconds between bytes before giving up. A value of 0s means this will never time out. Defaults to 60s, as per Vinyl Cache's default settings.

bind

Hostname and port on which Vinyl Cache will listen for requests. Defaults to 127.0.0.1:8000.

connect-timeout

If specified, this option configures the connection timeout (in seconds) for Vinyl Cache connecting to a backend server. Vinyl Cache will only try to connect to a given backend for this many seconds before giving up. Defaults to 0.4s, as per Vinyl Cache's default settings.

cookie-pass

This list consists of lines with a cookie-match and urlexclude in the form: "cookiematch":"urlexcludes". If cookiematch applies for the cookiename, then the request is passed directly to the configured backend bypassing any caching. Additionally, if the current url matches urlexcludes, then the cookies are removed, and the request piped to the backend. Defaults are optimized for Plone, one line: "auth_token|__ac(|_(name|password|persistent))=":"\.(js|css|woff|woff2)$" So when you are authenticated, the request is always handled by Plone. When an authenticated user requests a js/css/kss file, Plone will see you as anonymous because no cookies reach Plone.

cookie-pass-not-exclude

If url matches this regexp, cookie-pass exclude rules are skipped. This is useful for url like ++resource++zmi that requires authentication also for resources like js, css, ...

cookie-whitelist

When the cookie-pass is processed and does not match, this means you are anonymous, at least with the default cookie-pass settings. In that case, this whitelist is used to to sanitize cookie data on the request. Cookie data to be sent to the backend includes only cookies with the given names. Defaults are optimized for Zope2/Plone: statusmessages __ac _ZopeId __cp auth_token The __ac and auth_token cookies should not be needed, as they are already in the cookie-pass list, but they are here for safety in case you have customized the cookie-pass setting to not include them. If you have custom code that sets cookies and needs to read them in the backend, then you must add the cookie names to this list.

first-byte-timeout

If specified, this option configures the timeout (in seconds) for Vinyl Cache receiving the first byte from a backend. Vinyl Cache will only wait for this many seconds before giving up. A value of 0s means Vinyl Cache will never time out. Defaults to 300s.

purge-hosts

Specifies hostnames or IP addresses for purge ACL. By default localhost and the backends are allowed to purge. Additional allowed hosts are listed here.

purge-by-id

Enables collective.purgebyid-compatible purging: a backend response carrying an X-Ids-Involved: #uuid1#uuid2#...# header gets those ids associated with the cached object, and GET /@@purgebyid/<id> (from an IP allowed by purge-hosts) purges every cached object tagged with that id -- without needing to enumerate every cached URL variant of that content. Possible values:

• off (default): disabled.
• ban (also on): purges via a ban() matching the X-Ids-Involved header. No vmod required, works everywhere.
• xkey: purges via the xkey vmod's secondary-key support instead of a ban scan; more efficient on busy caches, but requires [varnish-build] compile-vmods = true (the xkey VCL import is only emitted in this mode, so off/ban never break compilation for setups that haven't built vmods).

max-cacheable-size

Objects whose backend response Content-Length exceeds this value are never cached (beresp.uncacheable = true). Unset (default): no size limit. Value must be a valid VCL BYTES literal, e.g. 50MB or 1GB. Without a limit, a single object near or above the cache's size can, under LRU pressure, evict a large fraction of the cache just to make room for itself.

verbose-headers

Enable sending extra diagnostic response headers (X-Cache, X-Cacheable, grace) that expose what Vinyl Cache did with the request and the cache status. Useful for debugging cache settings and optimizations; leave off in production to avoid exposing cache internals to clients. Possible values: on or off (default).

vcl_recv, vcl_hit, vcl_miss, vcl_backend_fetch, vcl_backend_response, vcl_deliver, vcl_pipe, vcl_purge, vcl_hash, vcl_import, vcl_init, vcl_pass, vcl_synth

Insert arbitrary VCL code into the generated config.

zope2_vhm_map

Defines a virtual host mapping for Zope servers. This is a list of hostname:ZODB location entries which specify the location inside Zope where the website for a virtual host lives.

zope2_vhm_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of Vinyl Cache, such as haproxy. Defaults to bind port.

zope2_vhm_ssl

If specified, this maps VHM URLs to https for all requests. Possible values: on or off (default).

zope2_vhm_ssl_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of Vinyl Cache, such as haproxy. Defaults to 443.

This recipe always generates VCL syntax version 4.1; there is no vcl-version option.

health-probe-*

Settings for backend health probes. Probes are activated if grace-healthy is set.

See the VCL reference on the Vinyl Cache documentation site for a detailed explanation of each setting.

• health-probe-url: defaults to /ok
• health-probe-timeout: defaults to 5s
• health-probe-interval: defaults to 15s
• health-probe-window: defaults to 10
• health-probe-threshold: defaults to 8
• health-probe-initial: If not given Vinyl Cache will default to threshold -1

grace-healthy

Grace in the context of Vinyl Cache means delivering otherwise expired objects when circumstances call for it. This can happen because: (1) the backend-director selected is down, or (2) a different thread has already made a request to the backend that's not yet finished.

If the backend is healthy, accept objects that are this number of seconds old. Clients will be delivered content that is no more than number of seconds past its TTL.

Format: number followed by a time unit: ms, s, m, h.

Defaults to None. If this is set to None the grace feature is disabled.

grace-sick

If the backend is sick, accept objects that are this old. See also grace-healthy.

Format: number followed by a time unit: ms, s, m, h.

Defaults to 600s. Should be greater than grace-healthy.

To test the generated configuration for syntactic correctness, run varnishd -C -f ./parts/varnish-configuration/varnish.vcl.

Create script to start Vinyl Cache

Start Vinyl Cache as a daemon or in foreground with the given settings. These options are available for the recipe part plone.recipe.vinylcache:script.

bind

Hostname and port on which Vinyl Cache will listen for requests. Defaults to 127.0.0.1:8000.

build-part

References the buildout part in order to get settings from there. Defaults to varnish-build. Set it to false in order to switch it off.

cache-location

Customise the location for the Vinyl Cache file storage. Option only applicable when used with the file cache-type option. Defaults to using a file named storage inside the relevant parts directory (eg parts/varnish/storage). Changing the default location can be useful in putting the storage somewhere with quicker read speeds (e.g. RAM disk).

cache-size

The size of the cache (limited to 2G on 32bit systems). Defaults to 256M.

cache-type

Specify the type of cache storage to use with Vinyl Cache. Possible values: file (storage for each object is allocated from an arena backed by a file), malloc (storage for each object is allocated with malloc; in memory), or other storage engines supported by your Vinyl Cache build (e.g. umem). Defaults to file.

configuration-file

Path to a VCL configuration file to use. Defaults to the generated file from the configuration-part setting. If no configuration was generated, this setting is mandatory.

configuration-part

Names the buildout part to get settings from. Defaults to varnish-configuration.

daemon

The file and path of the Vinyl Cache daemon varnishd to use (the installed binary is still called varnishd as of Vinyl Cache 9.0.x, despite the project/brand rename). If not given, it looks for the build part (see the build-part setting) and uses its location setting plus the string /sbin/varnishd. If there is no build part, it defaults to /usr/sbin/varnishd - the most common place where it's found on many Unix systems. Adjust it if needed.

group

The name of the group that Vinyl Cache should switch to before accepting any request. This defaults to the main group for the specified user.

mode

Specify whether the Vinyl Cache daemon should run in daemon or foreground mode. The latter is useful when Vinyl Cache is run by service supervision tools like daemontools or runit. Defaults to daemon.

name

Sets the name of the Vinyl Cache instance. Defaults to ${buildout:directory}/var/<this part's name>, not varnishd's own system default -- which is typically somewhere under /var/run and requires root, the single most common "Permission denied" trap when running varnishd unprivileged from a buildout. Deliberately not placed under parts (that tree is meant to be disposable/regenerated, e.g. whenever varnish-build is recompiled, which would lose this runtime state) and keyed by the part's own name so multiple Vinyl Cache instances in the same buildout don't collide. Set this explicitly if you want the traditional host-name-based default or a different location.

From varnishd's manpage:

Amongst other things, this name is used to construct the name of the directory in which varnishd keeps temporary files and persistent state. If the specified name begins with a forward slash, it is interpreted as the absolute path to the directory which should be used for this purpose.

runtime-parameters

Runtime parameter configuration options. The full list of available options can be found in the manpage varnishd(1) for your version of Vinyl Cache. Examples include thread_pool_max, thread_pool_min, sess_timeout.

telnet

If specified sets the hostname and port on which Vinyl Cache will listen for commands using its telnet interface.

tls-config

Path to a hitch-style TLS configuration file, mapping to varnishd -A (a Vinyl Cache 9.0 addition). Lets varnishd terminate TLS itself instead of needing a separate TLS terminator (e.g. Hitch, nginx, haproxy) in front of it. See the plone.recipe.vinylcache:selfsigned recipe below for a quick way to generate a matching self-signed certificate and config file for internal/dev/testing use. Not set by default (no -A flag is emitted).

script-filename

Name of the start script file in buildout:bin-directory. Defaults to the name of this buildout part.

secret-file

The telnet interface is not usable without authentication by default. A pre shared key mechanism has been put in place which requires both the Vinyl Cache daemon and a client connection over telnet (like the varnishadm tool) to have a shared key to authenticate. By default if no secret-file is specified, it's no longer possible to authenticate to the telnet interface.

To disable this security feature use secret-file = disabled. This is discouraged.

To enable the secret-file, give the path to a file on the filesystem that preferably has random content and is both accessible to the Vinyl Cache daemon and a command line utility like varnishadm.

An example buildout part to generate such a file could be:

[varnish-secret]
recipe = plone.recipe.command
command = dd if=/dev/random of=${buildout:directory}/var/varnish_secret count=1
          chmod 600 ${buildout:directory}/var/varnish_secret

Giving secret-file the location of this file will pass on the secret to the Vinyl Cache daemon when it starts up. Afterwards you can use varnishadm with the parameters -T host:port -S /path/to/varnish_secret to connect to the admin telnet interface.

user

The name of the user Vinyl Cache should switch to before accepting any requests. Defaults to nobody.

Generate a self-signed TLS certificate

plone.recipe.vinylcache:selfsigned generates a self-signed certificate/private key (via the openssl command line tool, which must be available on PATH) and a ready-to-use hitch-style config file, for pairing with the script part's tls-config option. Meant for internal/dev/testing use -- being self-signed, clients need to explicitly trust this certificate (or ignore validation errors); it is not a substitute for a CA-issued certificate on anything internet-facing.

Generation is idempotent: an already-present certificate/key is not regenerated (and so not rotated/invalidated) on later buildout runs.

Example:

[varnish-tls-cert]
recipe = plone.recipe.vinylcache:selfsigned
common-name = internal.example.org
bind = *:8443

[varnish-script]
recipe = plone.recipe.vinylcache:script
tls-config = ${varnish-tls-cert:config-file}

bind

Hostname (or *) and port the generated hitch-style config file's frontend block will listen on. Defaults to *:8443.

common-name

The CN (Common Name) of the self-signed certificate. Defaults to localhost.

days

Certificate validity, in days. Defaults to 3650 (10 years) -- long-lived since this is meant for internal/dev use where rotation ceremony isn't the point.

key-size

RSA key size in bits. Defaults to 2048.

key-file, cert-file, combined-file, config-file

Output paths for, respectively: the private key, the certificate, the two concatenated together (what the hitch-style config's pem-file directive points at), and the hitch-style config file itself (what you point tls-config at). All default to sensible locations inside this part's own parts directory.

Examples:

Use system Vinyl Cache at /usr/sbin/varnishd, generate start script in ./bin/varnish using a VCL-file in ./parts/varnish-configuration/varnish.vcl:

[buildout]
parts =
    varnish-script
    varnish-configuration

[varnish-script]
recipe = plone.recipe.vinylcache:script

[varnish-configuration]
recipe = plone.recipe.vinylcache:configuration

Credits

plone.recipe.vinylcache is a fork of plone.recipe.varnish, created to track the Varnish Cache -> Vinyl Cache project rename and the 9.0.x release line. All credit for the original design, the VCL generation approach and the years of maintenance goes to the plone.recipe.varnish authors and contributors.

If you don't need Vinyl Cache 9.0.x and are happy on Varnish 6.0 LTS, please use plone.recipe.varnish directly; it is unaffected by this fork and keeps receiving its own maintenance.