Plone 4: three times faster than Drupal, Joomla and WordPress

Plone 4 is about to leave its alpha testing phase and enter beta testing prior to a final release. Update: Plone 4 has made a final release; I’ve updated the graph.

One of the many things the Plone team has worked really hard on in this release cycle is improving Plone’s performance.   Plone core developer Hanno Schlichting has blogged about this a number of times, and deserves a tremendous amount of individual credit for digging deep into Plone’s innards to find and fix inefficiencies.

When I read Hanno’s most recent performance post, in which he shows Plone 4.0 alpha 3 serving up 22 pages/second without caching on his personal laptop, I  started to wonder how this compares with some of the other common CMS platforms out there.  I was pretty sure none of them could deliver more than 10 pages/second with zero caching or performance tuning.  So I did a bit of quick-and-dirty speed benchmarking.

I’ll start with results, then explain my methods.  As I suspected, Plone 4 is faster out of the box than some of the most common PHP platforms.  Lots faster.

That’s right:  Out of the box, Plone 4.0 served up 15.1 pages/second, that’s over three times faster than Drupal 7 alpha 1’s 4.1 pages/sec, Joomla! 1.5.15’s 3.6 pages/sec or WordPress 2.9.1’s 4.5 pages/sec.  Plone 3.3.6 our currently shipping release, turned in a snappy 9.4 pages/sec, over twice as fast as its PHP competitors.  Whee!

Testing methodology

Here’s how I tested.  Hardware was my 2GHz MacBook with 2GB RAM.  I installed Plone 4.0 using buildout, and used MAMP 1.7.2 to run the PHP products.  I did a default install of each product, no add-on modules, I used whatever default content and initial configuration each product provided.  (Joomla! gives you the option whether or not to install sample content, which I accepted.)  No caching was installed or configured for any system.   I measured performance of the homepage as an anonymous user with trusty ol’ ab -n 10, ran it a few times to get the systems warmed up, then noted the value where the runs stabilized.

Lies, damn lies and statistics

So, what does all this mean?  Well, honestly, not much.  (Although Hanno clearly has a faster laptop than I do!) This is obviously a crude benchmark.  I didn’t load up each CMS with realistic sample content.  I tested two pre-release products (Plone 4.0 alpha 3 and Drupal 7 alpha 1) against three production releases (Plone 3.3, Joomla! and WordPress).   All of these CMSes can easily pump out hundreds of pages per second with a little bit of tuning, reverse proxy and/or database caching (although many real-world users don’t bother with performance optimization!).  So, this benchmark is definitely not an accurate measure of the real-world performance of a site.  (Matt Hamilton recently explored this a bit.)

But I think that Hanno and the rest of the Plone team can be very, very proud of Plone’s raw speed.  And as Hanno points out, there are even bigger gains just around the corner; he’s targeting 50  pages/second (without caching), as his goal for Plone 5.

25 thoughts on “Plone 4: three times faster than Drupal, Joomla and WordPress”

  1. WordPress 2.9 out of the box, 4 pages per second ? Come on, be serious.
    I understand you wanna protect your slowly dying Plone, but lying in stats, that’s a nasty thing to do ;-)

  2. Spark – I published my methodology; you are free to repeat my experiment and see what results you produce, then tell me where I made a mistake.

  3. I suspect MAMP not to be very efficient on Mac OS X — but I may be wrong.
    Anyway, what is really interesting about this benchmark is that Plone 4 seems definitely faster out of the box than Plone 3. Which is encouraging as such.

  4. The test could be more eloquent by testing functunality like researsch internal pages with one thousand content published.

  5. @pierre: Plone 4 is going to be lots faster than Plone 3. Follow the links in the article to Hanno’s blog posts for lots more detail on that, if you’re interested. If you know more about why MAMP might not be a “fair” benchmarking environment, please do let me know. As a Plone guy, I’m not that well-versed in PHP app deployment.

    @enclope: I agree, it would be nicer to design a benchmark with standardized test harnesses of content, but that is far beyond my skill level and time availability. I’d love to see folks with interest in this topic design a suite of standard performance tests!

  6. Drupal has different caching options (e.g. block caching, page caching, CSS aggreggation, Javascript aggregation) but they are all disabled by default. In other words, looking at the out-of-the-box performance of Drupal 7 is only so useful.

  7. Thanks for stopping by, Dries. Yes, as I mention in the article, all of the CMS products mentioned have extensive, configrable caching options, and all are (rightly) disabled by default. A logical conclusion from this article would be that enabling caching for all of these CMS systems is pretty much required across the board in almost all realistic production situations. :-)

  8. I suspect the difference is down to Plone’s use of the ZODB. For read-only operations, once the objects have made it into ZODB’s object cache no database queries need be made. Object invalidation happens automatically behind the scenes. Contrast this to the PHP/MySQL systems which must perform several selects in order to render the page.

    Of course when you need to fetch new objects it’s slower, in fact I wouldn’t be surprised if Plone is quite a bit slower in this case as ZODB fetches are serial. Archetypes persistent object design is far from optimal (page data is split over several persistent objects, slowing down load time), which is being addressed as part of Dexterity.

  9. Hi there,

    I’d be interested if you were able to discover anything about the following:

    1. What % of the pageload effort was due to MYSQL and what percentage was due to Apache/PHP/Zope/Python? As CMS’s scale (ie — the tables of their database grow) the balance tends to shift, with more and more of the overhead moving from the application side and toward the DB side. I’d be interested in what this split was for each CMS.

    2. With caching disabled, what is the minimum number of queries required to generate a simple page? (It is in all cases greater than zero, and usually greater than 4-5 … but I’d be curious. Depending on the CMS and the structure of the page a blank page can be shockingly expensive to generate.)

    I love thinking about this stuff. Thanks for your post.

  10. @matt — great questions! I didn’t hookup any instrumentation to discover the answers to these questions. Doing so is definitely possible and a great idea, but way beyond my skill level. I’d love to see someone tackle this.

  11. @matt – Good questions, but one of the main differences here is the fundamental architectural difference between CMSes that use the RDBMS and some kind of ORM or manual SQL (like Drupal, Joomla) and those that use an OODBMS such as Plone. Plone has the advantage that it is much faster at page loading at the data level as it doesn’t have to go through an ORM, however as Laurence points out this can have downsides as many object fetches are done in serial.

    So it is actually very difficult to say ‘how much is the app and how much is the db?’ in a meaningful way across the different architectures. Plone uses Zope Enterprise Objects (ZEO) to scale, so you could do instrumentation on CPU load for the ZEO server (db) versus the ZEO clients (app) however you’d probably find again the numbers fairly meaningless as, in the case of ZEO, the ZEO server is designed to be very very simple, and uses very little CPU at all and is mainly disk bound.

    As for ‘queries to render a page’ again, this concept is fairly meaningless in the case of Plone as the underlying persistence layer takes care of all of this for you without you even knowing about it. What we can look at on the Plone side is the number of objects fetched to generate a page. This is somewhat analogous to how many rows would be retrieved from a RDBMS to serve a page in the relational model. One thing that we have been working on in Plone 4 is to reduce the number of object loads needed.

    @dries – Yes most CMSes have caches, Plone is no different, but we wanted to compare ‘out of the box’ speed, since that is something a lot of people historically have not liked about Plone. Yes it can go fast, but it always needed tuning. Now there is less tuning needed to start with.

    Another very important point is that caching is not always possible in some circumstances, e.g. authenticated users, portals, personalisation. Yes, you can cache blocks or page fragments etc, but it all adds complexity and has to make assumptions on how long data is going to be valid for, so needs to be configured on a per-usecase basis. Plone 4 has managed to optimise the actual core code so even when caching is not applicable we still get the performance boost.

  12. First I’d like to thank Jon for bringing this topic up. I’ve used to think default Plone install isn’t that fast. This changed a bit when I first time started Plone 4 – still after many years of Plone developing/admin the image of Plone speed is quite deep in my mind and doesn’t wear out easily :)

    I was interested about the comparison and especially about the huge differences so I did similar tests with more requests and some concurrency against WordPress and Plone. Plone still won, but added concurrency made WordPress perform much better. The difference between WordPress and Plone was only 3 requests / second in favor for Plone.

    When I tried same with Plone page which had navigation portlet visible, results were even more equal. With 100 request and concurrency set to 3 I got following results:

    WordPress: 13.04 requests / second
    Plone: 14.64 requests / second.

    I’m still glad to see Plone outperform it’s PHP rivals with out of the box experience – although difference wasn’t that big with added concurrency.

  13. @Matt Hamilton

    Thanks for the interesting info about Zope/Plone. As you can guess I come from the PHP/WordPress/ExpressionEngine world, so this is an education for me. I’m definitely do some more reading about this.

    However, as you hinted, I think it surely must be in some way possible to distinguish between the effort required to render a page itself, and the effort required to retrieve stored content. But I agree that the comparison would not be apples-to-apples for the architectural reasons you mentioned. (BTW one of the things I’ve found attractive in the little Zope-related reading I’ve done is that object storage seems to be highly abstracted to include things like memory-based storage … ie not just disk-based DBs. We can’t do that natively in many PHP-based CMSs)

    One thing that comes to mind: there is a difference between what I call vertical scaling (how fast can you serve a single URL) and horizontal scaling (how fast can you serve an array of 5000 different URLs) …. This is a significant problem for any CMS with or without caching. A CMS might be excellent at rapidly serving the same page over and over, and fail totally when presented with real-life traffic situations. (If I sound like I’m speaking from experience here … um well … I might be …)

    So I guess while I appreciate these sorts of comparisons between CMS’s, I think their value is somewhat limited. I’d rather have a 30% slower web application that scales horizontally than a lightning-quick CMS that has trouble with simultaneously generating many different pages in rapid succession. I guess it’s all about how the ap re-uses and regenerates whatever cache (be it object, query, page, whatever) it relies on.

    Anyway, more Random thoughts!

  14. @matt
    Big-scale content repositories like Grist’s are not unusual in the Plone world, and that’s one of the reasons why the Plone team has invested a great deal of effort in improving the “raw” speed of Plone. As you point out, folks with big repositories like grist have a pretty big “long tail” of people visiting old articles — and as you point out, that makes effective caching significantly more challenging and/or more expensive in terms of resources (e.g. you need a larger cache to hold more stuff). As a result, your visitors have a much higher likelihood of hitting uncached content.

    Which is actually why the “raw, uncached” speed of your CMS (which is what I am measuring in a very, very crude way here) may actually matter somewhat, despite the vital role of caching.

  15. Jukka,

    The speed benchmarks you post with concurrency are relevant too, but most production instances of Plone run with 2-N ZEO clients, which will help this a lot. What you’re testing now is a 2-thread (maybe 3-4?) Plone against a PHP app that creates/destroys the thread on every request, which does scale “wider” out of the box, if that makes sense.

    Even our installer sets up ZEO with clients for you, so this is “out-of-the-box”, but we don’t make decisions on load balancing for you. But correct, we start getting into more advanced setups then.

    Jon’s point was simply to counter the (incorrect) perception out there that “Plone is slow”, where in reality it’s actually as fast as or faster out of the box than Joomla/Drupal/Wordpress, which are by some considered to be pretty lightweight and fast.

    And I think he succeeded with that task.

  16. Very interesting article, an insightful read. Old perceptions from colleagues (and others) certainly do die hard though — even with such evidence.
    Here’s looking at you, Plone 4 & 5.

  17. Pingback: life « Yolanda

Comments are closed.