Plone 4 is about to leave its alpha testing phase and enter beta testing prior to a final release. Update: Plone 4 has made a final release; I’ve updated the graph.

One of the many things the Plone team has worked really hard on in this release cycle is improving Plone’s performance.   Plone core developer Hanno Schlichting has blogged about this a number of times, and deserves a tremendous amount of individual credit for digging deep into Plone’s innards to find and fix inefficiencies.

When I read Hanno’s most recent performance post, in which he shows Plone 4.0 alpha 3 serving up 22 pages/second without caching on his personal laptop, I  started to wonder how this compares with some of the other common CMS platforms out there.  I was pretty sure none of them could deliver more than 10 pages/second with zero caching or performance tuning.  So I did a bit of quick-and-dirty speed benchmarking.

I’ll start with results, then explain my methods.  As I suspected, Plone 4 is faster out of the box than some of the most common PHP platforms.  Lots faster.

That’s right:  Out of the box, Plone 4.0 served up 15.1 pages/second, that’s over three times faster than Drupal 7 alpha 1′s 4.1 pages/sec, Joomla! 1.5.15′s 3.6 pages/sec or WordPress 2.9.1′s 4.5 pages/sec.  Plone 3.3.6 our currently shipping release, turned in a snappy 9.4 pages/sec, over twice as fast as its PHP competitors.  Whee!

Testing methodology

Here’s how I tested.  Hardware was my 2GHz MacBook with 2GB RAM.  I installed Plone 4.0 using buildout, and used MAMP 1.7.2 to run the PHP products.  I did a default install of each product, no add-on modules, I used whatever default content and initial configuration each product provided.  (Joomla! gives you the option whether or not to install sample content, which I accepted.)  No caching was installed or configured for any system.   I measured performance of the homepage as an anonymous user with trusty ol’ ab -n 10, ran it a few times to get the systems warmed up, then noted the value where the runs stabilized.

Lies, damn lies and statistics

So, what does all this mean?  Well, honestly, not much.  (Although Hanno clearly has a faster laptop than I do!) This is obviously a crude benchmark.  I didn’t load up each CMS with realistic sample content.  I tested two pre-release products (Plone 4.0 alpha 3 and Drupal 7 alpha 1) against three production releases (Plone 3.3, Joomla! and WordPress).   All of these CMSes can easily pump out hundreds of pages per second with a little bit of tuning, reverse proxy and/or database caching (although many real-world users don’t bother with performance optimization!).  So, this benchmark is definitely not an accurate measure of the real-world performance of a site.  (Matt Hamilton recently explored this a bit.)

Andreas Jung recently wrote “zopyx.trashfinder,” a quick-and-dirty script for assessing whether folks releasing software via the Python community’s “PyPI” repository are providing minimal metadata and actually making releases.  (Andreas has been on the warpath against sloppy release management, it seems, perhaps having been recently frustrated.)

Andreas has assessed the zope., plone., and collective.* namespaces so far.  He found 2 of 167 zope.* products with metadata problems, 2 of 117 plone.* products with problems, and 15 of 325 collective.* products with problems.

I decided to run an assessment against the Products.* namespace, which also contains lots of Plone add-on products.  The news is pretty good… out of 250 products checked, only 4 had problems.

CRAP: Products.Clouseau==0.8.4dev - description < 40 chars
CRAP: Products.Clouseau==0.8.4dev - summary < 10 chars
CRAP: Products.Clouseau==0.8.4dev - no author and no maintainer email given
CRAP: Products.Clouseau==0.8.4dev - no author and no maintainer name given
CRAP: Products.listen==0.7.1 - description < 40 chars
CRAP: Products.listen==0.7.1 - no author and no maintainer email given
CRAP: Products.listen==0.7.1 - no author and no maintainer name given
CRAP: Products.LoginLockout==0.2 - no release files, no valid download_url
CRAP: Products.PloneInvite==1.1-alpha - no release files, no valid download_url

While I think Andreas is being a bit (characteristically) tough in his conclusion that “PyPI is the public data toilet of the Python community,” given the low rate of problems, I really love the idea of regular, automated sanity checking like this, or as Alex Limi and I have called it, “automated shame.”  I hope Andreas adds the product maintainer or author’s name to the listing in a future release.

I’d love to see someone regularly interrogate all of the packages in Plone.org’s PyPI server with this query, and regularly publish the results to the Plone product-developers list.

Update: David Glick also referred me to Matthew Wilkes’ “mr.parker” which checks to make sure that PyPI packages have more than one authorized admins, in order to avoid the “hit-by-a-bus” factor.  Good stuff.

Commenting is the building block of most “social” features on the web.  Plone has had commenting since its very early releases, and got a lot of things right the first time.  However, the web has become more sophisticated since 2004, and our “mostly good enough” commenting framework is now showing its age.  It’s time for a rewrite, and that rewrite is under way!  Here’s a short progress update, and an invitation to help out.

Thanks to Google Summer of Code 2009, Timo Stollenwerk is leading the work on plone.app.discussion, a modern, lightweight, easy-to-extend commenting system for Plone.   Martin Aspeli is serving as the project mentor, and is playing a large role in designing the architecture, project management and quality assurance — all with a goal of ensuring that we have code of sufficient quality to land in Plone 4 later this year.  I’m playing the role of the “customer,” helping to articulate and prioritize requirements and testing the results.

Check it out!

Our roadmap and progress log is public at http://www.pivotaltracker.com/projects/15135.  As you can see, we’re using an Agile development process on this, thanks to Martin.   Basic commenting functionality is now working, and we’ve got iterations planned out for the rest of the summer (and beyond!).  You can give our development version a try at http://gsoc.timostollenwerk.net/.  More experienced users can get their hands on the code with our development buildout.

I’ve been brainstorming a bit about how Plone do an even better job of helping non-technical website editors manage and optimize their images for the web, and I’ve come up with what I think is a pretty clever idea.  I have neither the time nor the skill to implement it, and so I thought I’d float it out there to see if

A) it holds water;

B) if I can’t interest someone in implementing it

The problem: big-ass images

Lots of people who edit Plone websites don’t really understand how to properly optimize images for the web via resizing and compression.   Plone is not entirely unhelpful here; if you upload an image, we automatically create several different resized versions for you to use in your pages.  However, Plone doesn’t currently do anything to help you make sure that original image you’re uploading is appropriately sized and compressed.

When you couple user inexperience with the massive image sizes that today’s digital cameras produce (6/8/10/12 megapixels!), the result is often a situation where users are uploading massive digital camera photos to their Plone sites, then using the PIL resizes in page.  While this works, it quickly produces a bloated ZODB.    In addition, the automatic PIL resizes don’t apply much compression, so the resized images aren’t very well optimized either.  This slows down site performance and causes lots of unnecessary server load.

There are a number of add-on products that can help in various ways, among them:

• Plone.app.blob and Filesystem storage, which push large binary objects such as images to the filesystem, reducing ZODB bloat.
• ImageEditor, the amazing Photoshop-like image editing program that runs inside of Plone.  It can do resizing and compression, but only one image at a time.  It’s a great solution for end-users, though.

These are great, but I think we can do better.

Some possible clever solutions

Here’s how I think we can do better.

First, some changes to Plone’s default behavior:

• Make the amount of compression that PIL applies to its resizes settable in the Plone control panel.  Right now it is hardcoded into ATContentTypes, and it’s hard to customize without rolling your own content types.
• Automatically resize all “original” uploaded images to 1024px.  Make it user-configurable, and disable-able for those rare occasions when you need to upload really high-resolution originals.

More daringly, I think that someone could write a clever add-on product for site administrators that would do the following:

1. Walk the site catalog, looking for all objects that are Image-ish and in JPEG format.
2. Look at the pixel dimensions and the filesize of each image.
3. Find images where the “bytes per pixel” (filesize/*(height*width)) is higher than a certain “reasonable” value.  (0.5 bytes per pixel seems about right to me, based on the idea that a 150X150 pixel JPEG image shouldn’t weigh much more than 10kb.  (Obviously, these settings should be user-editable.)
4. Present the user with a list of “suspiciously large” images, along with their pixel dimensions, filesize, bytes per pixel and a preview.
5. For the JPEG images, offer to apply additional compression to some or all of the selected images.  (Make value user-settable, maybe offer an opportunity to test it against 2-3 images, and estimate the savings you’d realize.)
6. For images that have larger pixel dimensions than anyone would want to display in a web browser, offer to resize the original image down to a user-specified size (1024 pixels seems like a reasonable default to me).
7. Apply compression via PIL, presumably in a background process so that Zope doesn’t bog down.  Rebuild Plone’s auto-resized images afterwards from this new original.  Purge caches as necessary.

So, whaddya think?  Is this sane?  Would you get value from these things?  Are there other low-hanging fruit I’ve overlooked?  Interested in helping roll the first two ideas into PLIPs for Plone 4?  Or building out the bulk image optimizer?  Or helping raise some money to sponsor such a project?  I’d love to hear from you.

The Plone community is hard at work on some game-changing innovations that will redefine your notion of what is possible with a content management system.  One of the coolest elements of this is Deliverance, a new system for “theming” (applying a custom visual design) to a Plone site.

Deliverance introduces the paradigm-shifting notion of “rules based theming” in which the theme consists of a simple set of rules expressed in XML which are capable of mapping chunks of a source page (raw Plone) into sections of a fully-styled HTML/CSS page.

It’s easier to see than describe: Nate Aune put together a fantastic set of slides about Deliverance for Plone Symposium East 2009, I can’t recommend them highly enough.

Even cooler bonus: Deliverance will let us build a new generation of “point and click” theming tools like Banjo, which Nate and Eric Steele whipped up during the post-Symposium sprint.  It’s a proof of concept, not ready for primetime yet, but it’s a pretty compelling glimpse into the future of theming.

This isn’t quite “mainstream best practice” for Plone yet, but it will be in the not-too-distant future.  I for one can’t wait!

One more thing:  Because Deliverance runs outside of Plone, you can use it to theme almost any web application!  I wonder if we’ll see Drupal and Joomla adopting Deliverance someday?

H/T to Elizabeth Zimmerman.

Geir Bækholt of Jarn just delivered a keynote talk at the European Plone Symposium 2009 in which he outlined the roadmap for the next two major releases of Plone.  You can skim through the slides here, but it’s worthwhile to click through to the full version so you can click on the “Notes” tab and read Geir’s notes to accompany the slides.

As of this week, and much to my surprise, I’m officially a published author!

Along with a dozen other talented Plone community members, I contributed three chapters to Practical Plone 3: A Beginner’s Guide to Building Powerful Websites.  A modest contribution the annals of technical nonfiction, to be sure, but I’m still tickled pink at the prospect of going into a bookstore and possibly spotting my name on the shelf.

I’m also really excited to have a beginner-friendly book on Plone 3 in the market.  I’m hoping that Practical Plone will ease folks’ climb up the learning curve to Plone mastery.  Enjoy!

I’m very excited about Products.TinyMCE, the new add-on product for Plone by Rob Gietema of Four Digits that integrates TinyMCE, the popular open-source graphical HTML, into Plone.

Background: Since Plone 2.1, released in 2006, Plone has shipped with an excellent graphical HTML editor called Kupu.  Kupu was a mature, capable product long before TinyMCE was a glimmer in its developers’ eyes, and Plone’s “batteries included” approach to bundling a high quality, cross-platform graphical editor is still a solid differentiating factor.

Of course, this being open-source, some people prefer alternative graphical editors.   There’s long been a decent integration of the unfortunately-named FCKEditor.  And the crew at OpenPlans have wrangled Xinha into Plone as well.  But, in my opinion, there’s never been a competitor to Kupu that equalled its features and improved on its usability.  Until now.

Here’s what I particularly like about Products.TinyMCE:

• It gets the “Link” drawer right.  Kupu has two separate drawers, one for internal (inside your site) links, and one for external links.  That’s confusing and awkward for users.  TinyMCE combines these into a single drawer, and raises Kupu by adding point-and-click support for mailto: and https: links and control over whether links open a new window.  The overall design of the drawer is a lot cleaner and it feels a lot more responsive.
• Similarly, Product.TinyMCE’s “Image” drawer is also a nice improvement.  Its features are basically equivalent to Kupu’s, although its support for image captioning is a bit cleaner.
• Product.TinyMCE’s support for creating and editing HTML tables is pretty solid.  This was always a weak area for Kupu, and a very difficult task for any graphical HTML editor.
• Products.TinyMCE handles Flash embedding in a clean and user-friendly way.  Flash embedding is possible in current versions of Kupu, but often feels a bit awkward and unreliable.
• TinyMCE is a lot easier to extend with custom code than Kupu.  At least that what David Glick tells me. 

Nothing is perfect, though.  Here are a few areas where Products.TinyMCE is not quite up to Kupu’s high standard.  Some I know are already on Rob’s list, so I’m hoping to see them in the next release soon.

UPDATE: TinyMCE 1.1RC is out, and addresses most of my suggestions below, except for Plone 2.x support. Try it, it rocks.

• Search support in the Image and Link drawers.  Kupu lets you search your site content in the insert Image and Link drawers.  That’s an incredibly powerful and useful feature, and it’s not there in Products.TinyMCE.  I know it’s high on Rob’s to-do list, so I’m sure this oversight will be remedied soon.
• Kupu has a nice feature where you can link directly to an anchor tag in a page your Plone site.  Products.TinyMCE can link to an anchor within the current page, but not to an anchor on another page.  This is not a feature I use often, but it can be handy.
• Kupu has a nice feature for automatically assigning anchor tags to heading styles, and managing them.  Products.TinyMCE doesn’t seem to have equivalent functionality.   Again, this is not a very frequently used feature, IMHO.
• Kupu works with both Plone 2.x and Plone 3; Products.TinyMCE only works with Plone 3.  That’s probably not a deal-breaker, but worth noting if you’re thinking about a switch.
• There’s a “save” button to save your draft without leaving edit mode; it works, but there’s no visual feedback to the user.

We’re about to start testing Products.TinyMCE with some of our users, to see if, like us, they think it’s more pleasant and powerful than Kupu.  If that pans out, we’ll likely start rolling it out across our Plone 3 projects.

And, one more thing…

One thing that would really help deployment of Products.TinyMCE on exisitng sites would be a script that could be triggered by a site administrator that would walk through all users and change their preferred editor to TinyMCE (or change it back to Kupu).  This could allow site admins to roll out a new editor without having to ask every user to manually change their preferences.   That would be a really great convenience that could really boost adoption.

Nathan Van Gheem has put out a 1.0 release candidate of ImageEditor, his fantastic image editing add-on product for Plone.  ImageEditor implements a slick, easy-to-use image editor inside of Plone.  ImageEditor lets Plone users quickly and easily resize, crop, sharpen and compress images they’ve uploaded without Photoshop.

A short screencast is worth a thousand words of description, so here you go: ImageEditor in 2 minutes.

Thanks, Nathan, for all the work you’ve put into making ImageEditor rock!  I can’t wait to start using it with all of our sites.

I’m eyeballing IntenseDebate.com, recently acquired by WordPress, who offer a hosted third-party blog/website commenting service, and am starting to wonder whether it might be a good idea to write a Plone add-on product that integrated IntenseDebate as a commenting system for Plone.

It seems like there would be some pros and cons.  What do you think?  Worth the effort or a waste of time?