Theming Plone websites with Diazo (for non-developers)

Plone 4.2 has just been released. Congrats to Release Manager Eric Steele and all of the excellent folks who worked hard to make it happen!  For me, this is one of the most exciting Plone releases ever, because this is the first release of Plone that includes the amazing Diazo theming system as part of the Plone core.

Diazo completely revolutionizes the process of theming a Plone website, and it’s already making the lives of everyday Plone integrators better  What’s more, while Diazo is now part of Plone, via the plone.app.theming module, it’s also a standalone website theming system that can be used the theme pretty much any website.  So, not only is Diazo a revolution in Plone theming, I also think that it has the potential to revolutionize how any website is themed.  (Diazo-powered Drupal or WordPress anyone? 😉 )

What is Diazo?

Diazo is a simpler and smart approach to theming a website–without sacrificing creative control.  Here’s how it works: Your graphic designer, who doesn’t need to know anything about Plone, builds you a theme consisting of one or more HTML + CSS templates.  Plone emits its HTML pages, known as the “content.”  Diazo then lets you merge bits and pieces of Plone’s HTML content into your theme’s HTML and CSS template.  Diazo’s merging behavior is governed by a simple text-based rules file consisting of simple directives such as <append>, <replace> and <drop>.  It’s a drop-dead simple syntax that anyone can master, and it doesn’t require you to understand anything about how Plone works–basic HTML and CSS literacy is all you need.

I’m intrigued, tell me more.

Diazo does its magic by building on top of XSLT (eXtenstible Stylesheet Language Transformations), a standard originally developed in the late ’90s that defines a language for transforming XML documents into other XML documents.  (HTML pages are one kind of XML document.)  Raw XSLT is too generalized and too complex for everyday use as a everyday website theming technology, so Diazo provides a simple and user-friendly set of basic rules that get compiled into XSLT behind the scenes.  If you’re already an XSLT wizard, you can use the full power of XSLT expressions in your Diazo rules files, but most folks will never need or want to do this.

Plone 4.2 comes pre-configured for Diazo (via plone.app.theming), so if you’re using Plone, you don’t have to lift a finger.  If you are using Plone alongside other web applications, or not using Plone at all,  most modern webservers already have support for XSLT transformations, so deploying Diazo via WSGI, Ngnix, Varnish or Apache is simple and straightforward.

Diazo’s been around for a while, and it’s been battle tested in production as an add-on product for several years now.  It’s ready for prime-time.  There are already a couple dozen Diazo-powered Plone themes available for download, and I’m expecting to see lots more as Diazo becomes “mainstream best practice.”  It’s pretty easy to take any generic open-source HTML/CSS website theme and adapt it for Diazo.

Sounds great, what’s the catch?

There isn’t one.  That’s why Diazo is so exciting.  One thing to think about, though, is whether Diazo’s definition of “theming” matches yours.  Unfortunately, if you’ve been using certain other CMSes, you might have some odd ideas about what’s part of a “theme.”  (Hint: not an e-commerce system!)  With Diazo, theming is about HTML, CSS and Javascript–it’s not about adding new logical functionality to your website such as image sliders, calendars, shopping carts or content types.  Sure, these kinds of features can be an important part of a website’s “look and feel” but they’re not legitimately within the scope of a Diazo theme.

So, if you need to make a Plone website look beautiful, Diazo is all you need.  If you need to fundamentally change what content Plone is emitting in the first place, then you’ll need to dive into Plone customization.  Diazo doesn’t completely replace all of the customization that you might to do Plone, but it narrows the scope of what you need to do inside of Plone quite a bit.

The future is even brighter

If you’ve got a vivid imagination, you might already be thinking, “OK, simple text-based rules files are cool, but what about a graphical editor for building themes?”  Well, that’s in the pipeline.  Martin Aspeli is putting the finishing touches on a version of plone.app.theming that includes a graphical editor for Diazo rules files.  This is currently slated to ship with Plone 4.3, and will make it point-and-click easy to build or modify Diazo theme rules.

Final thoughts

I’m really excited about Diazo.  It’s the fulfillment of a vision for “rules-based theming” that’s been percolating in the Plone and Python communities for a number of years now, starting with Paul Everitt’s original implementation of Deliverance way back in 2008.  It was a radical idea then, and it’s still cutting-edge now.  Diazo brings rules based theming into the mainstream, both for Plone and for other web applications.  I think we’re going to look back a few years from now and wonder why we ever themed websites any other way.

Hidden gems of the Plone Collective

Periodically, I trawl through the Plone Collective repository (both via Github and via PyPi) to see what folks are building but not publicizing widely via Plone.org.  As usual, I found some hidden gems that I think deserve a bit wider attention.

Fair warning: I’ve tested each of these products in a Plone 4.1.5 development instance, but I’ve not deployed any in production or reviewed the code.  All are written by experienced Plone community members, though, so they should be at least reasonable sane.  I’d love to hear about your experiences with any of them.

collective.folderorder – Jens Klein and BlueDynamics Alliance

http://pypi.python.org/pypi/collective.folderorder/1.2

Plone’s default folder view shows items in the order they were added, and while you can manually rearrange items, there’s no way to automatically resort items in a folder view.  (Although you can add a collection as the default view of a folder, this is not always obvious to new users, and quite a few clicks.)  With collective.folderorder, you get a new “Order” option the Actions menu for a folder, and you can easily choose from several default folder ordering schemes, including: reverse order, unordered, and partial ordering. Even better, it provides an easy way for developers to add new ordering schemes.

I’d love to see this one PLIPed for future inclusion in Plone, possibly with a few more ordering options (e.g. last modified, creation date). It’s a small but welcome UI affordance.

collective.folderpositionLaurence Rowe

http://pypi.python.org/pypi/collective.folderposition/1.0

Another small but welcome improvement to folder ordering.  This one adds a nice little set of buttons below a folder listing that allows you to move items instantly to the top, the bottom or up/down a designated number of slots. Again, super convenient when you need to rearrange a lot of folder items.

collective.prettydate – Franco Pellegrini  & Héctor Velarde

http://plone.org/products/collective.prettydate

collective.prettydate reformats the display absolute dates/times (e.g. 3/1/2012) to relative date (e.g. one month ago, four days ago, etc.)  This is really nice for news sites or sites with upcoming events.

collective.embedly – Quintagroup

http://plone.org/products/collective.embedly

collective.embedly makes it stupidly simple to embed almost any externally hosted multimedia (YouTube, Vimeo, Slideshare.net, etc.) into Plone.  It uses the fantastic service “Embedly” which is itself built on the open “oembed” standard.    Developers who want a bit more power may also want to look at collective.oembed.

collective.routes – Nathan Van Gheem

http://plone.org/products/collective.routes

This one is a bit conceptual, but pretty awesome.  From the prolific and talented Nathan Van Gheem comes collective.routes, which makes it possible to build URLs in Plone that do catalog queries, e.g. http://mysite.com/blog/date/of/blog/post.  This isn’t really an end-user product, but it makes it easy for integrators to build really nice URLs for their custom Plone apps.

Game-changer for low-cost Plone hosting

Wow. Long-time Python community hosting favorites WebFaction just upped the memory on their low-end hosting plan to 256MB (not including OS memory) for $5.50/month.  You can bump it to 512MB for another $7/month.  That’s more than enough memory to comfortably run a modest Plone site, and when you combine it with WebFaction’s solid support reputation, suddenly you’ve got a fantastic low-end Plone hosting option.  Nice.

Chameleon page templates make Plone 4 another 20% faster

Plone 4 out of the box is already one of the fastest content management systems around.  And it’s still getting faster.  Malthe Borch has cut a major new release of Chameleon that gives Plone another big speed bump with almost zero effort.

Chameleon is a lightning fast drop-in replacement for the Zope Page Template system that Plone (and other Python applications) use.  By simply adding Chameleon to your Plone 4 buildout, you can immediately speed up your site by 20% or more, with zero code changes.

The homepage of a stock Plone 4.1rc3 instance yields 13.23 pages/sec on my laptop test environment[1].  With Chameleon added to the buildout, it jumps to 15.94 pages/sec.  That’s a 15% increase.  Not too shabby!  In the real world, where your page templates are doing considerably more work, you should see an even bigger boost.

Malthe’s been brewing this up for a while now, and it’s great to see it cross over into “ready for everyday production use.”  Give it a try!  Malthe’s taking bug reports if you run into any hiccups.

[1] 2GHz MacBook, 2GB RAM, Plone 4.1rc3 via svn, no proxy caching.  ab -n 20 -c 3, average of 10 runs

Check out Cook Inletkeeper’s awesome new “Weather & Tides” feature

At the end of last week, we pulled the trigger on the new inletkeeper.org for our friends at Cook Inletkeeper in Homer, Alaska.   It’s almost certainly the last major website I’ll launch as a staffer at Groundwire, and while it’s definitely a little bittersweet, I couldn’t be more proud of the results here.  In many ways, it’s a pretty typical “state of the art” website for a small conservation organization.  Plone makes that pretty easy these days.  But the trick I’m most proud of, both strategically and technically, is the “Weather & Tides” feature.

The staff at Cook Inkeeper dreamed up this “engagement superpower” in the course of living and working in, on and around the waters of Cook Inlet.  In Alaska, the weather is big, just like everything else.  And so are the tides.  And while there’s a ton of information about the weather and tides available online, it’s pretty scattered across different sites, and there’s no “single source” that pulls together all of the “must have” information for Cook Inlet residents.  That’s where we came in.

Inletkeeper staffer Michael Sharp, himself an avid sailor and surfer (when he’s not rocking campaign & communications strategy), identified the various sources for terrestrial and marine weather forecasts and current conditions data, and showed us a really cool iPhone app for generating tide predictions.   We took a look at the data sources–yep, all easy-to-parse RSS and XML.  And even better, it turned out that the tide prediction software underneath the iPhone app was open-source!

A few hours of development time later, my colleagues Matt Yoder and Ryan Foster had a slick, open-source and open-data powered “Weather and Tides” page for the new inletkeeper.org.  Then Cook Inletkeeper asked “Hey, can you make it work great on the iPhone, too?”  Matt got his mobile-fu on, and managed to “mobilize” the page with a few clever bits of CSS and Javascript.  No custom app required, this is all straight-up HTML.  I particularly like the way you can “swipe” to move between panels.  It’s hard to tell this isn’t a native iPhone app.

A few other nice details we managed to work in:

  • The tide graph also includes a perpetual tide table, formatted to look just like Cook Inletkeeeper’s popular print tide tables booklet.
  • Cook Inletkeeper staff can point-and-click to edit the list of weather and tide stations shown.
  • The page uses cookies to automatically remember your customized settings.
  • Cook Inletkeeper staff can automatically “hot-link” to custom settings, great for customized email marketing outreach.
  • “Recommend on Facebook” button

One of the things I will miss most about Groundwire is the thrill of being able to help conceive and create fun, effective tools like that help environmental groups engage their audiences.

Live “Google Docs”-like editing in Plone (and real-time chatting too!)

Yiorgis Gozadinos from Jarn recently released some amazing new work last week, a suite of packages called jarn.xmpp.* that enable powerful real-time interactivity in Plone, including:

  • Real-time simultaneous document editing (that’s right, just like Google Docs!)
  • Instant messaging/status updates between Plone users (think Twitter or Yammer)

Yiorgis has a great little video that shows this in action. Pretty magical.

collective.contentstats: nice!

Long-time Plone community member Raphael Ritz just released collective.contentstats, a new little add-on for Plone that generates a simple grid view that counts up each of your Plone content objects by content type and by workflow state.

In addition to the obvious function of providing a quick and easy way to track how many pieces of content you’ve got in your site, being able to quickly click through any number to get a list of the matching content objects is a tremendous simple way for site managers to zero in on draft content that might have gotten abandoned before being published, or old, retracted content that is ripe for deletion.

We’re already using it on Groundwire.org, and I’m looking forward to rolling it out to Groundwire clients.  Thanks, Raphael!

Introducing Diazo and plone.app.theming

Laurence Rowe and Martin Aspeli just made the first beta releases of Diazo and plone.app.theming.  I couldn’t be more excited.  Diazo is a revolutionary new approach to theming web applications.  And, while it has been developed by the Plone community, it’s not just for Plone.  Diazo can be used to theme almost any web application, and I think it heralds a paradigm shift in how we think about theming websites.  Diazo works best with Plone 4.1 (in beta, coming very soon),  which includes its dependencies, but can also be installed in Plone 4.0.x.

I can’t wait to see what folks will be able to accomplish with Diazo.

 

Three new Groundwire sites

We must have been feeling the Earth Day vibes, because my colleagues at Groundwire have launched three new Plone-powered websites in the last week:

Cascadia Center for Sustainable Design & Construction — the greenest commercial building in the world, currently in planning by our friends at the Bullitt Foundation.

Mountains to Sound Greenway Trust — leading and inspiring action to conserve and enhance the landscape from Seattle across the Cascade Mountains to Central Washington, ensuring a long-term balance between people and nature.

Harvesting Clean Energy —  a program of Climate Solutions, that helps accelerate rural economic development in the Northwest through clean energy development.

It’s great to be able to work with folks getting the good work done.

All good things…

After 15 years, I’m leaving Groundwire.

I’ll be starting a Masters in Public Administration at the University of Washington’s Evans School of Public Affairs this September.  I’ll wrap up my work at Groundwire in June and take the summer off to be a full-time dad to Everett and to enjoy summer in Seattle through the eyes of a fifteen-month-old.

I’ve been unbelievably fortunate to be a part of Groundwire over the past decade and a half.  I’ve learned a ton, worked for hundreds of amazing, inspiring environmental organizations and have been blessed with the most kick-ass colleagues and co-conspirators this side of anywhere.  I am more grateful to all of you (past and present) than I can ever adequately express.  Thank you.

I won’t be going too far away.  We’re staying here in Seattle.  I’ll continue to serve on the boards of the Plone Foundation and Green Media Toolshed.  It’s possible I’ll add a consulting gig or two to my plate once I get a handle on my academic workload.

While there’s no denying that this feels like the end of a huge chapter in my life, it also feels like a new beginning.  I’m really excited to plunge into the unknown and into what I hope will be a period of creative uncertainty.  While I don’t know what the next chapters looks like, I’m confident that they will remix familiar themes: public service, social change, openness, systems thinking, data-driven decision-making and smart use of technology.

Watch this space for further updates.  Be seeing you.

 

 

Migrating WordPress from Apache to Cherokee

[Sorry, this is gonna be a geeky one.  You’ve been warned.]

For the past couple of years, I’ve had this blog on a 256MB VPS slice at Rackspace Cloud, which, overall, has been a very nice ~$12/month experience.  But I’ve chafed a bit at having to restart my Apache web server instance every so often, because it has run out of RAM.  I’m just running a personal blog, goshdarnit.  How can that burn up 256MB of RAM!?!? Turns out the culprit is Apache.  Or, more accurately, the combination of Apache + PHP.  PHP prevents Apache from running in its more memory-efficient “worker” configuration, and the result is that even a site with almost no traffic can easily run out of memory.  Worse, tuning Apache to keep memory usage low means that it starts to perform really poorly.  Bottom line: it’s actually really hard to run a simple WordPress (or other PHP apps) behind Apache on a virtual private server with limited memory.

I’d been casting about for a solution, and had almost settled on using Ngnix, the increasingly popular open-source web server that lots of my friends in the Plone community really love.  But, truth be told, I was a bit intimidated by configuring it to work with PHP, even though it’s far simpler than Apache (which I also barely understand).

Then, David Bain turned me on to Cherokee.  Cherokee is a new-ish open-source web server that is designed to combine the speed of Ngnix with insane ease of configuration + deployment via a user-friendly point-and-click web administration panel.  And, sure enough, it didn’t disappoint.  In about an hour, I was able to install Cherokee, configure PHP, point Cherokee at my WordPress instance, and migrate a few rewrite rules to handle my WordPress shortlinks.

My site is now cranking out over 100 pages/second while RAM usage is well under the maximum and we’re never swapping into virtual memory. Suddenly, the WordPress editing interface feels reasonably responsive.  Vroom!

Looking into the future, I’m pretty excited about the potential to easily deploy Plone behind Cherokee.  Cherokee has built-in, easy-to-configure uWSGI support, which means that we’ll be able to start messing with Plone 4.1 + Cherokee.  (Plone 4.1 will ship with Zope 2.13, which is the first Zope 2 release to ship with unofficial WSGI support.)  This will remain “experimental” for a release or two, until the community’s had time to explore & document best practices.

Vote for your favorite Plone Conference 2010 talks

As usual, there are way more fantastic Plone Conference 2010 talk proposals than slots.   That means it’s time for you to help out Matt Hamilton and the rest of the Plone Conference 2010 team at Netsight by voting for your favorites.   There’s just a couple of days left: voting ends on Sept. 17th, and the selected talks will be announced on Monday, Sept. 21.  Gonna be a long weekend in Bristol. 😉

5 Things That Rock About Plone 4

The Plone community, led by Plone 4 Release Manager Eric Steele, just pushed out Plone 4, a huge upgrade to my favorite content management system.  Incorporating over three years of work from over 300 contributors, there’s a lot to be excited about.  Plone 4 takes Plone to new heights of speed, usability and just plain good looks.  Here at Groundwire, we’ve been developing several client projects against pre-release versions of Plone 4, so we’ve already had a few months to put some mileage on it.  Here’s what I’m most excited about:

1) New graphical HTML editor: TinyMCE

For most day-to-day Plone users, the graphical HTML editor *is* Plone.  Plone has shipped with a graphical HTML editor included since Plone 2.1 in 2005, and it still amazes me that some major open-source CMS projects don’t include one out of the box.  (Yes, I know some users like “choice,” but that’s no excuse for not having a sensible default.)

Plone 4 now includes TinyMCE as its graphical HTML editor, replacing Kupu, which has served us long and well.  (And which continues to be an optional add-on.)  TinyMCE deliers a much more polished, user-friendly and customizable content editing experience

I waxed enthusiastic about TinyMCE when it was first released as an add-on for Plone 3, and my love for it has only grown deeper with time.  Our not-so-HTML-savvy clients love it, too.

2) New default theme: “Sunburst”

For the first time in its 9-year history, we’ve shipped a new default theme with Plone.  It’s called Sunburst, and it’s a triumph of elegant minimalism in web design.

Sunburst’s beauty is more than skin-deep.  Designer (and Plone co-founder) Alex Limi is User Experience Lead for Mozilla Firefox, so he knows a thing or two about how to make the web beautiful, usable and accessible.  Sunburst incorporates all of the wisdom he’s gained over the years, and a TON of usability feedback from everyday Plone users, Plone integrators and usability experts.

The other great thing about Sunburst is that it’s easy to customize.  We’ve stripped down the templates and the CSS, removing a lot of clutter that had crept into the default Plone templates over the years.  This will make things much easier for site integrators and theme builders and sets the stage for more improvements to “through the web” theming in Plone 4.1.

3) Vroom!

Plone 4 is fast, fast, fast.  If there was a singular obsession of the core Plone developers during the Plone 4 release cycle, it was making Plone go faster than ever before.  Hanno Schlichting was the “speed champion” for Plone 4.  While most of the Plone team has been focused on building new features and refining our UI, Hanno was tinkering in the lab, doing performance profiling to figure out exactly where we were wasting precious milliseconds.  Then, he dove into the code and figured out where to make exactly the  right tweaks to speed things up.

The results speak for themselves. Plone 4 is about 50% faster overall than Plone 3.  It handles heavy user loads far more gracefully (up to 200% better than Plone 3).  It uses less RAM.  And our “out of the box” uncached performance is now over 3x faster than the default installs of PHP solutions like WordPress, Joomla! and Drupal that are often regarded as “good performers.”

And the best is yet to come.  Plone 4.2 is planning to offer another optional 50% performance boost by letting you use Chameleon, a drop-in replacement templating engine that works with our existing Zope Page Templates.  (Brave folks can try out Chameleon right now, but you have to be certain that all of your templates are XHTML-compliant.)

Hanno’s goal for Plone 5 is to boost our performance by another factor of 3 and hit 50 pages/second uncached on commodity hardware.  Vroom!

4) BLOBs: how unfattening!

In olden days, Plone stored uploaded files (aka “BLOBs” — that’s Binary Large OBjects) directly in the ZODB.  That was clean and simple, but unfortunately, meant that sites with massive numbers of documents & images produced bloated ZODBs that could be hard to manage.  Worse, Zope used to be somewhat inefficient at serving up these big files, which could really slow down a site.

There have long been add-on products for Plone such as FileSystemStorage that alleviated these problems, but these were imperfect solutions.  As it turns out, the fix really lies at the level of the Zope database itself.  And Plone 4 uses ZODB 3.9, part of Zope 2.12, which includes native support for filesystem storage of BLOBs and for super-efficient uploads & downloads.   This is a big deal, especially for Plone-powered intranets, who will find their ZODB sizes and RAM usage dropping dramatically and user satisfaction zooming skywards.

(Bonus: Plone 3 users can take advantage of this technology by using plone.app.blob, although getting it installed and configured does require solid intermediate Plone-fu.)

5) More bling!

Plone 3 included jQuery, the awesometacular javascript library that now handles most of our javascript interactivity.  Plone 4 ups the ante by including jQuery Tools, which makes it easy to build super-polished user interfaces with modal popup dialogs and forms.  We take huge advantage of this in Plone 4, by moving lots of commonly-used forms and confirmation dialogue boxes into fast, slick popup forms.  This makes day-to-day tasks like logging in, deleting content, and adding users faster and more enjoyable than ever.

Plus, add-on product authors and themers can now count on jQuery Tools being there, letting them build even more polished and effctive user interfaces for their add-on code.  For example, we’re taking advantage of it in Megaphone to make a really slick wizard for composing online advocacy letters and petitions! Plone.org’s sliding panels and accordions are all built with jQuery Tools.

Coming soon: external link tracking (and more) in Plone with collective.googleanalytics

Matt Yoder here at Groundwire is brewing up a really nice new feature for his already-excellent Google Analytics integration for Plone, collective.googleanalytics: tracking of external links, file downloads and mailto: links via Google Analytics “event tracking.”

If you’re not already familiar with collective.googleanalytics, that’s OK — it’s still in beta. 🙂  But it’s definitely worth checking out.  Collective.googleanalytics builds on the basic Google Analytics integration capabilities that are built into Plone already.  By taking advantage of Google Analytics’ web services API, collective.googleanalytics pulls selected snippets of your live Google Analytics data back into your Plone site so that site managers get “in their face” analytics data as they are managing content.  We find this really useful with our busy nonprofit clients, who might otherwise not be paying as much attention to their analytics as they ought to.

As we’ve been exploring the intersection of Plone and Google Analytics, we’ve realized that there is a bunch of interesting data that Google Analytics doesn’t automatically capture because they don’t generate full-fledged pageviews for the basic Google Analytics javascript to detect.  For example, “out of the box” Google Analytics doesn’t track PDF or other binary file downloads, people clicking on outbound links, or clicks on non-http links like “mailto:”.  (This isn’t Plone’s fault, it’s a consequence of Google Analytics’ fundamental design architecture.)

However, Google does provide a powerful, flexible (and fairly new) Events Tracking feature that you can call to track interesting non-page-view events like these.    And that’s what Matt is taking advantage of in his upcoming release of collective.googleanalytics to increase the amount of data that Plone sends to Google Analytics.

The collective.googleanalytics control panel has now grown a new set of configuration options:

new google analytics settings

As you can see, you now have the option to enable or disable tracking of email (mailto:) links, external outbound links, and file downloads.  Even better, these extensions are pluggable, so that if your site needs to track more kinds of events, you can easily write small plugins and have them appear in this menu!  Check here for samples of plugins, which should be easy to adapt for your own needs.  Matt would love contributions of additional useful tracking plugins!

Once configured, results will start showing up in Google Analytics under Content>Event Tracking, like so:

collective.googleanalytics is in beta right now; we’re using it on Groundwire.org and on a  few of our clients’ sites — to rave reviews thus far.   A “1.0 final” release should come sometime in the next few weeks. Matt’s new work on external link tracking is still in a branch, and will be merged soon.

Bonus points:

  • Matt’s new branch also branch generates the basic Google Analytics tracking javascript for you automatically using the profile you select in the tracking section of the configlet.  (No need to copy-and-paste the javascript snippet from Google anymore, like you do in out-of-the-box Plone.  That’s a nice convenience feature.)
  • All of these new features use the new asynchronous Analytics tracking API, which  should offer some performance benefits over the old blocking javascript, which could sometimes slow down your page loads.

If you’re a Plone developer or integrator, we’d love you to check out, kick the tires and offer feedback.   General comments here are great — we also have a bug tracker on Plone.org for bug reports or feature requests.

More thoughts on automated image optimization for Plone

Last June, I wrote up some initial thoughts about how Plone could to more to help folks with limited experience preparing images for the web.    It got quite a bit of favorable response, but then I went off on sabbatical and haven’t followed up, until now.  I’m more convinced than ever that this is a good idea, and I really want to build some energy and resources around it.

Here’s some updated thinking on what I think could be done, both in core Plone and in a new add-on product, along with a simple wireframe and some implementation notes.

Making core Plone smarter

Plone 4 includes a frequently-requested feature that makes images much smarter: through-the-web configurable image scales.  Previously, these were hard-coded into ATContentTypes.  Now, we have a new “Image Scales” control panel, and site admins can add, edit and remove image scales.  This makes it much easier to provide the image scales site users need, and to eliminate the ones they don’t.  Nice work!

There’s one more change that I think should happen here:

Let’s make the amount of JPEG compression that PIL applies be easily configurable.  Right now, it’s hardcoded into our zope.conf, and it’s really, really high.  When we prep images for the web here at Groundwire, we typically use a compression quality of 60.  I think that it would make a lot of sense to expose this variable through-the-web in the Image Scales control panel configlet.

Rough Spec for an “Image Optimizer” Add-on Product

Ok, now here’s where the cool stuff starts.  Here are my updated thoughts on an add-on product that I think would really make life easier for sites that have lots of poorly-optimized images.

I imagine an “Image Optimization” control panel configlet that will present a UI that allows the user to:

  1. Walk the site catalog, looking for all objects that are Image-ish (i.e. have an image field) and contain images that are in JPEG, PNG or GIF format.
  2. Look at the pixel dimensions and the filesize of each image.
  3. Find images where the “bytes per pixel” (filesize/*(height*width)) is higher than a certain “reasonable” value.  (0.5 bytes per pixel seems about right to me, based on the idea that a 150X150 pixel JPEG image shouldn’t weigh much more than 10kb.  (Obviously, these settings should be user-editable.)
  4. Present the user with a list of “suspiciously large” images, along with their pixel dimensions, filesize, bytes per pixel and a preview.
  5. For images with dimensions that are larger than user-specified value, offer to resize them.
  6. For images that are too “heavy per pixel,” offer to apply more compression (for JPEGs) or convert to JPEG (for GIFs/PNGs)
  7. Apply changes via a separate process (via plone.app.async) with AJAX updates so that Zope doesn’t bog down.
  8. Afterwards, offer to rebuild Plone’s auto-resized images afterwards from this new original.  Purge caches as necessary.

Here’s a wireframe of what I’ve got in mind:

A few more implementation notes/questions:

  • Alec Mitchell recommended using ImageMagick, because it offers much better quality than PIL.   I think this also seems sensible, because it would offer an easy way to spawn the intensive image manipulation work off into a separate process.  An outside-of-Zope processing approach such as plone.app.async is probably worth looking at as a model.
  • Resize, then convert, then compress.
  • JPEGs should be compressed via ImageMagick.
  • PNGs could potentially be compressed by running through OptiPNG.
  • GIF’s can’t be compressed
  • Use AJAX to refresh the screen on scan and as optimization proceeds.  Show optimization results by crossing out weight/dimension/density and replacing with new values.  (Hide unselected images when you hit Optimize.)  Total up savings at the bottom when optimization is complete.
  • Can Plone’s image scale rebuild be run in the background via plone.app.async so we don’t tie up a Zope process?
  • Plone 4 support is required; Plone 3 support would be very nice and would probably greatly increase the value of this product.
  • For extra bonus points, warn the user of unoptimized images right after upload, and offer to fix it “on the spot.”

Somewhat randomly, I also wanted to take note of Kurt Bendl’s useful recipes for limiting the max KB and pixel dimensions of uploaded AT Images and Files.  This might be useful and/or worth exposing in a TTW setting.

Next steps

Doing a mediocre job on this would probably be pretty easy, but it will take some focused effort to really nail the details that will make this sing.  I think this is a project worth raising a little bit of money around, and hiring someone good to really own it.

If your clients would benefit from a tool like this, I encourage you to think about whether you can raise some cash from them for a bounty fund.  I don’t have a hard total in mind just yet, but I’m thinking that $3000-$5000 might be a reasonable range.

I’ve got a couple of people in mind that I’d like to approach as possible developers on this, but if this description strikes a chord with you and you’ve got both the passion and skills, let me know!

This also might make a great Google Summer of Code project, in which case I’d love to supplement that by paying an experienced Plonista to be a strong, hands-on mentor and ensure that the project gets done and done right.  (This worked really well with Martin Aspeli and Timo Stollenwerk on plone.app.discussion!)

I’m willing to PM this and be “the customer” and Groundwire can very likely help seed the bounty fund as well.  Let’s make this happen!

Popup Forms for Plone

Shortly before dashing out the door for Pycon 2010, David Glick pushed out a 1.0 release of Popup Forms for Plone, which he and Steve McMahon built on top of Steve’s excellent Pipbox and PloneFormGen products.

Popup Forms for Plone makes it point-and-click easy to create timer-driven javascript popup forms anywhere in your Plone site.   You can see a simple example in action at Washington Conservation Voters.   It’s amazingly simple: just build your form in PloneFormGen.  (If you just want to popup a static HTML page or an image, you can use PloneFormGen’s “Form Prologue” and skip adding any form fields!)  Then, you use Plone’s portlets mechanism to assign the form to a page or folder on your site, and to configure an optional time-delay.  That’s it!  No programming, no javascript, no fuss, no bother.

Popup Forms are great for email capture forms, action alerts, user surveys, and many other calls-to-action.  If that’s what you need in your Plone site, I encourage you to check it out!