I’ve spent some time setting up varnish for a service at work where we deliver JSON-fragments from different resources as a unified document. We build the resulting JSON document by including the different JSON fragments through an ESI (Edge Side Includes) statement in the response. Varnish then fetches these fragments from the different services, allowing for independent caching duration for each of the services, combining then into a complete document.
The main “problem” is that of Varnish 2.0.4, varnish does a check to see if the content returned from the server actually is HTML-based content. It does this by check if the first character of the returned document is ‘<'. The goal is to avoid including binary data directly into a text document. Since we’re working with JSON instead of regular HTML/XML, the first character in our responses is not a ‘<', and the ESI statement did not get evaluated. You can however disable this check in varnishd by changing the esi_syntax configuration setting - just add the preferred value to the arguments when starting varnish. To disable the content check, add:
varnishd … -p esi_syntax=0x1
I’m not sure if you can set this for just one handler / match in varnish as this solved our issue, but feel free to leave a comment if you have any experience with this.
I’ve done the same thing om /etc/default/varnish, but must be quick to add that I only configure ESI processing of the URIs I’m sure I want it, testing the URIs in vcl_fetch.
On a related note, Varnish doesn’t (yet) support ESI processing of gzip-ed content, so you must put a (lightweight) HTTPd in front of Varnish if you want to deliver gzip-ed content that has been ESI processed. I’ve both used Apache and Nginx for this.