Why WP Rocket fails on high traffic sites and how to fix it

We are big users of WP Rocket (licensed through 2020). WP Rocket works great on our clients' normal busy sites (up to 5 million visitors/month). We run one very high traffic network with up to 40 million visitors in one week or even a million visitors/hour.

On this site, WP Rocket fails miserably. When new articles are posted WP Rocket purges the cache of too many different pages:

• all the category archives to which the page belongs
• related date archives
• even the previous post and previous post in the same category (this is to keep the prev/next links on the posts working, if the post is updated it clears the next post too)
• feeds
• and of course the homepage

And all of the above happens if a post is added, updated or a comment is posted. So when you have people writing new comments every second you end up with a lot of cache purges on the pages where the information about new comment is not so important - new comments should only purge that post cache and nothing else.

Also - the full page cache is purged anytime a new user registers!

So we ended up implementing a queue for the cache purge actions. That way we are able to limit the amount of purges of any given URL to once per 10 seconds or even 1 minute for the less important pages.

Another problem is that once the cache entry for any given URL is purged, it has to be visited again to make sure it's in cache. On our heavily optimized site with PHP 7 + MariaDB + Redis and all sorts of fragment caching this takes 200 - 300 ms.

So if you have hundreds of visitors coming in every seconds, these requests come in and backup the pipeline, clogging our powerful server like a drain. All of them start to generate the cache file at once resulting in a disaster.

The proper fix for WP Rocket would be to not purge the URL by deleting it and then hoping the first visitor will re-generate the cache entry, but generating a new version of it on background and storing that in cache. That way at no point would visitors then be facing a missing cache and the server would not overload. 

Of course we couldn't leave a site to crash like this so we had to configure NginX Microcaching (fastcgi_cache). It took us some time to perfect our caching rules for it, but it's really fast and prevents race conditions altogether.

It's a pity though that WP Rocket isn't tuned for really high traffic as the benefits would be huge. Specially if it's free competitor WP Super Cache has the solution - check it's "Cache rebuild" option - it makes sure a supercache file is served to anonymous users while a new file is being generated. Perhaps we should give that one a try.

As we have a pro video plugin (WP.org) and a baker's dozen free plugins to take care of and we are generally happy with WP Rocket, we have no plans to take caching into our own hands. I sincerely hope that Jean-Baptiste and Caspar will add this improvement, at least as an option, for high traffic sites. We did report the issue to WP Rocket GitHub repository but got no answer for 5 months now.

Alec Kinnear

Alec has been helping businesses succeed online since 2000. Alec is an SEM expert with a background in advertising, as a former Head of Television for Grey Moscow and Senior Television Producer for Bates, Saatchi and Saatchi Russia.