Open Source CMS Security, Part II

Last summer, I did a quick count of the number of known security vulnerabilities in common open-source CMS products, and their underlying software stacks. The results were rather eye-opening.

I thought it might be time for an refresh. Once again, my protocol was simple: I searched the MITRE CVE list of known vulnerabilities and counted the number of results.

Here are the most recent results, with last July’s results in parenthesis for comparison, followed by the percentage growth rate:

  • Plone: 3 (3) – 0%
  • Drupal: 55 (22) – 150%
  • Mambo: 91 (31) – 194%
  • Joomla!: 74 (20) – 270%
  • Zope: 16 (15) – 6%
  • MySQL: 129 (99) – 30%
  • Python: 18 (17) – 5%
  • Rails: 2 (0) – infinite
  • PHP: 2271 (1258) – 80%
  • Ruby: 14 (7) – 100%
  • Perl: 105 (97) – 8%

Again, Plone, Zope and Python come out with remarkably low total issue counts and extremely low rates of new issues being found. Perl also seems doing pretty well, with relatively few new issues being found. Rails is also looking pretty good.

The rate of growth in new PHP vulnerabilities is still pretty staggering, both in absolute and percentage terms.

I’m also surprised to see the number of vulnerabilities in Drupal, Mambo and Joolma! continue to soar. (Joomla! 270%! Ouch!) It’s worthwhile to note that many of these vulnerabilities (but not all) are in add-on modules rather than the core products, and so may reflect more on individual module developers than the platform as a whole. Still, the fact that these products’ security exposures are growing considerably faster than that of their underlying PHP/MySQL frameworks is intriguing.

Again, in the end, these data don’t really prove anything, but they certainly are an interesting metric to keep an eye on over time.

I don’t think most folks choosing CMS platforms (or programming languages/frameworks), either as customers or as developers, are really considering the security track records of different tools. Should they?

11 thoughts on “Open Source CMS Security, Part II”

  1. Actually, this was one of the main reasons why we chose Zope and Plone for our CMS for Reefs.org and AdvancedAquarist.com. The last thing we want to be doing is patching up security holes when we could be doing other more important things. 🙂

    I agree, folks really should include security as a very important criteria when selecting a CMS. Other factors certainly apply of course but it should definitely rank up there on the scale.

    Shane

  2. This is a really good item for people to think about when looking at open source… security. The problem with statistics like this is they can mean almost anything:

    (1) Does Joomla/Drupal take security more seriously than Plone because they invest the time to find more security issues? Is Plone more secure than Joomla/Drupal because they all invest equally in searching for security flaws, and fewer are found in Drupal?

    (2) Do the building blocks of a system cause flaws in that system? Does Python have fewer security flaws than PHP because fewer people are searching for flaws in Python? Does Python have fewer security flaws because it is inherently better?

    (3) Are proprietary systems any better? Do we even have any idea what the flaw rate is for something like Salesforce?

    In the end, I think the comparison of these things is just silly and generates the worst kind of fear, uncertainty and doubt among lay people that are not qualified to/ nor should have to judge the relative technical merits of complex software.

    IMHO
    (1) All these systems/ approaches are of a very high quality. You will not go wrong selecting Plone, Joomla or Drupal.
    (2) All these programming languages are fine. Enterprises use them, universities use them.

    Finally these factors are virtually irrelevant in comparison to the most important factor:
    (1) Do you have someone your can trust and is qualified to implement/maintain your system? If Plone gets implemented by One Nowthwest, it will be implemented correctly. If it is implemented by an inexperienced college student, it is likely to be insecure. Same for Joomla and Drupal.

  3. David-

    You are of course correct that these statistics don’t mean much, and could be interpreted lots of ways. (I make that point myself.)

    One interpretation I don’t consider valid, though, is that the high rate of vulnerabilities being found in PHP and applications that us it is simply the result of “more eyes” looking for bugs.

    That’s like saying that Windows is just as secure as Mac OS X, it’s just that more folks are looking for bugs in Windows. There are indeed many more eyes on Windows, and more bugs being found, but there are also some fundamental differences in the underlying architectures that have led many to conclude that the fundamental engineering choices of Mac OS X make it inherently more secure to start with.

    Now, of course, Windows is generally secure enough when deployed intelligently. (I use it myself, and don’t feel uncomfortable with it.) But the fact that Mac OS X is a lot more secure “out of the box” at a minimum means that there are a lot fewer security mistakes for inexperienced users to make in the first place, like forgetting to use antivirus software.

    I suspect there’s a similar phenomenon going on with these programming languages and frameworks. I think that some of these platforms have more problems with security because there are more easy mistakes to make. Perhaps the fact that some frameworks have more inexperienced developers writing products add-on modules is also a factor.

    (BTW, I suspect that CiviCRM is in pretty good shape because the developers are very experienced folks, and there is strong technical leadership.)

    I do believe that the number of security vulnerabilities does say something about the quality of the underlying software. Of course it’s not the only measure or even the most important measure of software quality, but it is a measure.

    I see remarkably little discussion of security our community (both “laypeople” and developers/integrators), and I hope that by highlighting the underlying statistics, it will spur greater awareness and conversation. I’m heartened to see a session at next-week’s Nonprofit Software Dev Summit. (http://devsummit.aspirationtech.org/index.php/Security_Checkup_for_Nonprofit_Software_Developers)

    For example, in 2006, we had to apply a couple of Plone/Zope security hotfixes to over 90 client sites. It was easy for us to do this quickly without requiring full-on version upgrades for each site. Rolling out point upgrades (2.5 -> 2.5.x) has been a bit more work, and I would like to see that get easier.

    Drupal appears to have had six security-related point releases between May 2006 and January 2007 (4.7.0-4.7.6). Joomla! has had 4 security-related point releases in the past two years. (And a bunch more “stability/bug-fix” releases.) What have the experiences of mass hosting shops like CSOD been like w/r/t deploying these security updates?

    Finally, I do indeed believe that people who are choosing and using software (of any kind) DO need to consider the security implications and risks of their choices. That is hard. But nobody else is going to do it for them.

    I don’t consider this topic “silly” nor do I think that talking about security is FUD. I’m sorry if you feel that way. Drupal, Joomla, and Plone are all good products overall. Most people working with experienced developers to deploy and maintain them will be fine. I hope that people read these statistics and ask themselves (and their trusted advisers) more questions about security.

    The people I worry most about are the people who self-implement these products without experienced help. I do think they are experiencing risks that they aren’t well equipped to understand or manage.

    The worst situation of all is where short-term volunteers implement a software solution, and then leave a client with neither the knowledge or ability to maintain it over the long run. I shudder to think about how many “orphaned” and unpatched Plone 2.0.x, Drupal 4.5.x, Joomla 1.0.x, and WordPress 2.0.x installations are out there.

  4. I don’t think most folks choosing CMS platforms (or programming languages/frameworks), either as customers or as developers, are really considering the security track records of different tools. Should they?

    In my experience, the lifecycle of successful FOSS tools goes something like this:

    1. Developer hacks something together over a weekend to scratch own itch
    2. Developer posts it on Web, people start using it
    3. Eventually enough people see it and use it that Developer has to generalize it for common use…
    4. … which attracts even more users …
    5. … who all have pet features they want added, which Developer adds
    6. Steps 4-5 repeat until product has large enough audience to catch attention of GNAA wannabes, who poke around and find a million holes
    7. Developer hastily patches holes, and learns some lessons about security

    In other words, I don’t think I’ve ever seen a FOSS product that was 100% secure on first release. What I look for personally is how the team behind the product reacts when a vulnerability is discovered — do they care? Do they warn their users? Do they patch it promptly? A team that does these things is likely to also be proactive in finding and quashing exploits before they hit the wild.

    Examples. My favorite open source CMS is Typo3, which nobody else in North America seems to have heard of. Last December there was a serious XSS vulnerability discovered in one of the core components. They announced it on their web site, had an interim patch ready to close the hole within 24 hours of the announcement, and had a new point release with the fix built in shortly afterward. That indicates they care about security.

    Contrast this with Opera Software, which rolls out security patches without telling anyone, which is a sign that they don’t.

  5. From the Joomlasphere, I thought I’d give my $.02 here. I ran a query on Joomla as well at this site and found something interesting: not a single of these security bugs are related to the latest stable version of Joomla 1.0.12.

    Yes, when you have more add-ons than any other community, you’re bound to get a lot of novice coders developing components that are buggy and have security holes. That’s why we’re working hard in the Joomla community to educate users to know what’s a well-developed component look like. As you can imagine, it’s a hard feat to accomplish.

    Meanwhile, maybe you should turn your counter on Joomla from 74 to 0. =)

    Cheers!
    -Ryan

  6. Are these open active issues, or issues that were addressed and closed. If the raw number is high, but they have been addressed, then it does not mean a thing. What matters is how long it takes to plug the hole.

    Strange to see that Plone is rated highly, when less than a week ago, ubuntu switched from Plone to Drupal. One of the reasons given was the security track record of Drupal, and a bad security incident with Plone that they experienced. Read more on Matt Nuzum’s blog.

  7. Khalid –

    Matt doesn’t specifically say that whatever they experienced with Plone was security related.

    The facts show that Plone has a stronger security record than most other open source CMSes. That doesn’t mean that it’s had no security issues, only fewer, and less serious.

  8. Khalid,

    I actually followed up on this a bit with Plone co-founder Alexander Limi, who told me:

    “First, the entire premise of this article is that they are leaving MoinMoin, not Plone. They used Plone to initially launch their website(s) back when Ubuntu/Canonical started in 2004, and it was done because it was the only thing that could get them up and running in a matter of days. The Plone site was always meant to be temporary.

    The reason I know is because I worked for them at the time, so they did indeed choose Plone because they had people who knew it back then. 😉

    This was also Plone 2.0, a very different Plone than the later versions, and none of the systems (including Drupal) were very sophisticated at this point.

    The fact that they couldn’t/wouldn’t reevaluate Plone after 3 years of additional development is disappointing, but not surprising. People tend to treat software like static entities, and if you’ve had bad experiences with an earlier version, you rarely go back and try it again later.”

  9. There are found-and-fixed issues. But my point is that systems with lots of found-and-fixed issues is still worse than systems with fewer found-and-fixed issues. Why? Because fixed issues means frequent, required security updates. Which a lot of less-skilled admins can’t keep up with.

  10. Your query is highly misleading! I know you didn’t claim it was scientific, but I am surprised you would present this as fact. Of course Drupal, Mambo, or PHP are going to return a lot of results because A LOT of people are building 3rd party components on top of them.

    Look through the joomla listing, only 3 entries MIGHT apply to 1.0.12 Stable (the current version).

    If anything its comparable to the systems you’re pitching as surprisngly problem-free.

  11. Only about six months late to this discussion, but being unfamiliar with some of the FOSS CMS apps and being asked to evaluate a copy for work security was my first concerning and I stumbled across this site. I have a couple questions and statements if you happen to get notification of updates to this thread.

    First, a quick correction:

    “That’s like saying that Windows is just as secure as Mac OS X, it’s just that more folks are looking for bugs in Windows. There are indeed many more eyes on Windows, and more bugs being found, but there are also some fundamental differences in the underlying architectures that have led many to conclude that the fundamental engineering choices of Mac OS X make it inherently more secure to start with.”

    Well, considering XP post SP 2 and Vista have a lower rate of vulerabilities discovered despite more eyes does indicate that they are more secure, though the inverse (fewer eyes, fewer vulnerabilities) doesn’t tell you much. The myth of OS X security is a fairly dangerous one for OS X users to subscribe to, especially in light of the fact that third party vulnerability trackers (say, Secunia) don’t validate it (worse, the time to update after disclosure is much longer, at least in 2007).

    On the specific topic at hand, my experience with FOSS applications of a similar vein (in this case forum software) it seems the vast majority of issues are actually zero days that are discovered AFTER a successful attack. PhpBB is particularly prone to this considering its relative popularity. In light of that, a certain significant percentage of vulnerabilities can be linked to market share, as the greater the market share the increased scrutiny from the wrong elements. Do you happen to know the relative popularity of each of the CMS listed. Plone seems to have a decent popularity, but being a bit unfamiliar with this specific type of product, is it widely regarded as the underdog in terms of usage?

Comments are closed.