Last modified: 2014-08-27 03:22:33 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T71460, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 69460 - Resource loader ignores requested caching directives / not caching properly
Resource loader ignores requested caching directives / not caching properly
Status: NEW
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: High normal with 1 vote (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-08-13 08:31 UTC by metatron
Modified: 2014-08-27 03:22 UTC (History)
7 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description metatron 2014-08-13 08:31:24 UTC
When loading a gagdget or custom script from eg. meta/mediawiki/enwiki with

-mw.loader.load('//meta.wikimedia.org...&maxage=86400&smaxage=86400')

the responded cache-control directive is 
- private, s-maxage=0, max-age=0, must-revalidate

which prevents these scripts from being cached properly and causes a lag (40-500ms) on each call.

(IMHO JS pages should have a reasonable default > 0)



Example with requested cache:

Remote Address:xx.xx.xx.xx:443
Request URL:https://meta.wikimedia.org/w/index.php?title=User:Hedonil/Test/XTools.js&action=raw&ctype=text/javascript&maxage=86400&smaxage=86400
Request Method:GET
Status Code:304 Not Modified
Request Headersview source
Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en,no;q=0.8,nb;q=0.6,vi;q=0.4,nl;q=0.2,zh;q=0.2,ru;q=0.2,en-US;q=0.2,zh-CN;q=0.2,zh-TW;q=0.2
Connection:keep-alive
centralnotice_bucket=0-4.2; uls-previous-languages=%5B%22en%22%5D; mediaWiki.user.sessionId=4n8HMszYkP6cFIWMpzFAXdt0yQt1KsLK; metawiki-campaign=loginCTA; metawikiUserID=4326783; metawikiUserName=Anthelix; metawikiSession=d8f8b1cff708d4d092ddc67b62df397d; centralauth_User=Anthelix; centralauth_Session=9057b1467cc6fa35e848cc313af5f71c; forceHTTPS=1
Host:meta.wikimedia.org
If-Modified-Since:Wed, 13 Aug 2014 04:18:39 GMT
Referer:https://en.wikipedia.org/wiki/Charles_Schild
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Query String Parametersview sourceview URL encoded
title:User:Hedonil/Test/XTools.js
action:raw
ctype:text/javascript
maxage:86400
smaxage:86400
Response Headersview source
Accept-Ranges:bytes
Age:0
Cache-Control:private, s-maxage=0, max-age=0, must-revalidate
Connection:keep-alive
Content-Encoding:gzip
Content-Type:text/javascript; charset=UTF-8
Date:Wed, 13 Aug 2014 07:55:42 GMT
Last-modified:Wed, 13 Aug 2014 04:18:39 GMT
Server:nginx/1.1.19
Vary:Accept-Encoding
Via:1.1 varnish, 1.1 varnish, 1.1 varnish
X-Cache:cp1065 miss (0), amssq50 miss (0), amssq31 frontend miss (0)
X-Content-Type-Options:nosniff
X-Varnish:948365503, 2378009697, 4022640906
ConsoleSearchEmulationRendering


Example without requested directive:

Remote Address:xx.xx.xx.xx:443
Request URL:https://en.wikipedia.org/w/index.php?title=MediaWiki%3AGadget-HotCat.js%2Flocal_defaults&action=raw&ctype=text/javascript
Request Method:GET
Status Code:304 Not Modified
Request Headersview source
Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en,no;q=0.8,nb;q=0.6,vi;q=0.4,nl;q=0.2,zh;q=0.2,ru;q=0.2,en-US;q=0.2,zh-CN;q=0.2,zh-TW;q=0.2
Connection:keep-alive
Cookie:centralnotice_bucket=1-4.2; uls-previous-languages=%5B%22en%22%5D; mediaWiki.user.sessionId=2QWfgI0PCzO7nqVHetUbLsTGj025kox1;enwikiSession=a2cd738feeaa6203a52e18e27710a479; centralauth_User=Anthelix; centralauth_Session=9057b1467cc6fa35e848cc313af5f71c; forceHTTPS=1
Host:en.wikipedia.org
If-Modified-Since:Mon, 23 Jun 2014 19:15:43 GMT
Referer:https://en.wikipedia.org/wiki/Charles_Schild
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Query String Parametersview sourceview URL encoded
title:MediaWiki:Gadget-HotCat.js/local_defaults
action:raw
ctype:text/javascript
Response Headersview source
Accept-Ranges:bytes
Age:0
Cache-Control:private, s-maxage=0, max-age=0, must-revalidate
Connection:keep-alive
Content-Encoding:gzip
Content-Type:text/javascript; charset=UTF-8
Date:Wed, 13 Aug 2014 07:55:42 GMT
Last-modified:Mon, 23 Jun 2014 19:15:43 GMT
Server:nginx/1.1.19
Vary:Accept-Encoding
Via:1.1 varnish, 1.1 varnish, 1.1 varnish
X-Cache:cp1053 miss (0), amssq55 miss (0), amssq54 frontend miss (0)
X-Content-Type-Options:nosniff
X-Varnish:2605122819, 917135542, 2533692991
Comment 1 Michael M. 2014-08-13 08:52:39 UTC
I can reproduce this on any WMF wiki (including *.beta.wmflabs.org, both logged in and logged out), but not on my private test wiki or on other public wikis like http://www.explainxkcd.com, where Cache-Control is set as requested by the parameters (or as the set default). So I guess it's Varnish that replaces the Cache-Control with "Cache-Control:private, s-maxage=0, max-age=0, must-revalidate"
Comment 2 Michael M. 2014-08-13 09:10:43 UTC
I can confirm that appending &bcache=1 to the URL will set the Cache-Control as expected, so this definitely is caused by http://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/ce1f2b638ab2a09097ffe207ec6dc35c3823bc1b
Comment 3 Andre Klapper 2014-08-15 12:48:43 UTC
(In reply to Michael M. from comment #2)
> I can confirm that appending &bcache=1 to the URL will set the Cache-Control
> as expected, so this definitely is caused by
> http://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/
> ce1f2b638ab2a09097ffe207ec6dc35c3823bc1b

Thanks for investigating.

Mark: Any comments, as you worked on that change?
Comment 4 Andre Klapper 2014-08-23 19:46:53 UTC
Mark: Any comments, as you worked on that change?
Comment 5 Greg Grossmeier 2014-08-25 15:59:41 UTC
(In reply to Andre Klapper from comment #4)
> Mark: Any comments, as you worked on that change?

That's an old change (2013) so he might have forgotten ;)

Krinkle/Roan/Trevor: Thoughts here on the right cache settings?
Comment 6 Mark Bergsma 2014-08-25 16:06:59 UTC
That change simply puts the same behaviour in place that was there with the Squid cluster before Varnish.

The problem is that we don't have separate Cache-Control headers for _our caches_ vs _clients_ in MediaWiki, and never have had. So MediaWiki's Cache-Control header is targeted at our caches, and Varnish replaces that header with one that for most URLs doesn't allow caching (just like our Squid config did). Unfortunately, Varnish can only guess from the URL.
Comment 7 metatron 2014-08-27 03:22:33 UTC
If this is caused by this lines (as Michael M. stated):

sub vcl_deliver {
   /* Override the Cache-Control header for wiki content pages */
   if (req.url ~ "(?i)^/w(iki)?/.*"
       && req.url !~ "^/wiki/Special\:Banner(Controller|ListLoader)"
       && req.url !~ "(?i)^/w/(extensions/.*|api\.php)"
       && req.url !~ "(?i)bcache=1") {
       set resp.http.Cache-Control = "private, s-maxage=0, max-age=0, must-revalidate";
   }
}

Why not just simply add a regex to meet the directives of the request?
Request-url and query-string are available at present; the "new" parameter bcache is respected, why not maxage/s-maxage?
IMHO at least "private" should be removed for requested .js/.css pages or for action "raw". This would allow proxies/local browsers to intelligently set caching. "private" + "must revalidate" leaves no elbowroom for systems to guess the right things.

https://meta.wikimedia.org/w/index.php?title=User:Hedonil/Test/XTools.js&action=raw&ctype=text/javascript&maxage=86400&smaxage=86400

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links