Single Stacks, or Network-Centric Web Services?

Reading Zack Rosen’s assertion that building applications inside Drupal-the-framework makes more sense than loose integration of complementary applications triggered some thinking that’s been rattling around in my head for a while.

I think that the next few years are going to bring tremendous challenges for applications that do not easily communicate with other applications that are “outside their platform” i.e are written using a different language/framework, run on a different server, etc.

I think the most powerful path forward over the long haul is internet-based integration between great applications that were designed from the ground up to allow for it.  In other words, web services APIs are going to become increasingly more important, and the particular application frameworks less so.  This the “small pieces, loosely joined” model, to echo the phrase that others have appropriated from Dave Weinberger’s influential book. There are some great communities and significant resources behind very cool projects that provide great functionality that I really want to be able to tap. I don’t want to have either persuade them all to develop in a single platform (it’s just not going to happen) or try to duplicate all of their functions in whatever platform I’m most comfortable in. (Which, truth be told, is “none of them.”)

My colleagues here at ONE/Northwest and I would much rather focus on integrating best-of-breed applications that have strong web services APIs and are designed around the assumption that external applications are first-class citizens of their ecosystems. (Damn, that’s a lot of buzzwords.)

At the end of the day, why should I have to care if an application is based on Python, Zope, PHP, Rails, Django, or some technology I’ve never heard of? Why should I have to run all my applications off a single server? That’s not scalable. We now have a whole set of standards and technologies to let applications communicate with each other over the internet.

“Web services” is one of those complex, slippery terms that means lots of different things to lots of different people. To me, in this context, it means applications that share data with other applications over the internet. The more of your application’s guts it can expose to the outside world, the more powerful your web services API.

Some applications that I think are really moving in the right direction with web services support are:

  • Democracy In Action — powerful API, alas, not yet well documented. Little known fact: the smart guys at Enfold Systems have releaesd a Python wrapper for the Democracy in Action API, which (supposedly) makes integration with Plone possible. Haven’t tried it yet myself. But I’m looking forward to it.
  • Salesforce.com. Holy cow, these folks really get it. I’ve heard that half of their traffic is through their web services API. This is how a relationship management database should be — accessible by most any external application.
  • WhatCounts. These guys do our email blasting. Lots of folks do email blasting, some probably just as well as WhatCounts. But what sets WhatCounts apart from the pack fo us is the fact that they have strong APIs. This lets us do cool stuff like pull in names from Salesforce, or inject new subscribers from Plone, or pull in content from a Plone site. (Well, technically pulling in content from the outside doesn’t use their web services API. But the point is that WhatCounts can pull in data from outside and let other apps push data in.)
  • Another, less strictly “web services” example is Plone’s new PlonePAS framework. Basically, it’s a framework for authenticating users and retrieving user data from any old data source you’d care to write a plugin for. We’re going to try to use it to integrate Salesforce.com and Plone.
  • The whole open-source GIS software ecosystem, most especially including MapServer. My next-door neighbor, Chris Davis of CommEn Space, has shown me some really mindblowing stuff with maps that dynamically draw in data from all over the internet, thanks to open data standards and web services.

Can you see an advocacy software ecosystem here yet? I can.

And let’s not forget all the “Web 2.0″ applications out there that are getting so much hype these days. One important thing about the most exciting of these tools such as Flickr and Del.icio.us is that they can be written to and read from by outside applications via web services APIs.  (Amazon has done amazing stuff here, too, albeit without getting much “Web 2.0″ credit for it.) This is where it starts to get cool. The days of monolithic application stacks that try to do everything are fading fast. A new “network-centric” software ecosystem is starting to bloom.

And the best part: nobody has to “deeply partner” or adopt a single platform to make it work. They just have to focus on building great web services APIs so that other applications can meet them halfway. That’s not easy, but it’s surely easier than getting everyone to adopt the same platform.

Some software tools that I really hope build strong web services APIs as they roll out their next releases include:

  • Green Media Toolshed
  • CiviCRM (their web services API work seems to have stalled out in favor work on a PHP API that only talks to a PHP application (like Drupal) installed on the same server. Hopefully their focus will soon return to playing with the outside world, too.)
  • Custard Melt
  • Advokit
  • All of those nonprofit online donation tools that I am too tired to list right now. You know who you are.
  • And, yes, Drupal, too. ;-)

I’m probably overlooking some other apps that ought to be listed here. Feel free to suggest them. It’s late.

19 thoughts on “Single Stacks, or Network-Centric Web Services?”

  1. Yes! This is exactly what Anders, Sky and I have been talking about recently, and we even presented a lighting talk at pycon this weekend on this very topic (notes to follow).

    One buzzwordy way to think about this is that its about creating mashup-able applications.

    Another way to put this is to take seriously the proposition that the internet is a platform, or even further, that its your container. If you push this analogy you can imagine creating both session beans, as well as entity beans – ie stateless as well as stateful web services. When you begin to imagine little apps w/ their own data storage the power of mashups expands greatly.

    This is my favorite recent example of this:
    http://radar.oreilly.com/archives/2005/05/new_way_of_doin.html

    But this entire site shuould serve as inspiration:
    http://programmableweb.com/

    Take another peek at this post – http://blog.ianbicking.org/little-apps-instead-of-little-frameworks.html

    and the docs behind tasty
    http://tasty.python-hosting.com/wiki/TastyRestInterface

    all made super-easy by this little cherrypy/turbogears filter that REST enables your objects:

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/444766
    (and a corresponding rest_invoke library for easily calling these methods).

    If we get our act together, the wsgi/paste stuff might even be able to make these types of services installable via eggs, with perhaps enough intelligence to short circuit the full http request if there is a local entry point offering that service.

    Apps like this are tough to distribute, but if you are already hosting and running applications for clients this approach could really make it easier to create a lot of variety in your offerings.

  2. Jonah,

    Thanks for all the links! Very helpful. It will take us all a while to adjust to this new way of thinking and building. But I think the potential is tremendous.

  3. Just a minor correction. All the CiviCRM api’s are exposed both via our SOAP layer as well as php function calls. CiviMail talks to CiviCRM exclusively via SOAP (since its actually amavisd doing the talking to the CiviCRM db)

    I don’t think its got to be one or the other. Depending on the situation / integration potential a php api layer is beneficial in some cases while in many a case xml integration works just fine.

    lobo

  4. Lobo-

    I apologize for the inaccuracy. However, my source was the CiviCRM web site, linked above, which said “The RPC API interface should be in place for Version 2.0″ and then went on to discuss the difficulty of implementing it. So, it seemed to be saying that a SOAP API was on the table for the future (awesome!) but not there currently.

    If you can provide a link to better information for CiviCRM’s web services API I’d appreciate it.

    PHP integration is certainly beneficial and “works fine” in a case when you want to integrate with another PHP application running on the same server.

    My point is that a PHP API layer is a very small window through which to see the outside world. It means you wind up trying to force everyone to develop on your platform of choice (PHP) rather than remaining platform agnostic and focusing on interoperability.

  5. Here’s an updated link to the updated SOAP documentation:
    http://objectledge.org/confluence/display/CRM/SOAP+Access+to+CiviCRM+API

    I think they key is having an open API layer to an application. You can expose them in multiple ways (internal php api’s, a soap interface etc). Our SOAP layer uses the same PHP api’s (we need a wrapper for each api call primarily due to the way the SoapServer library works), and we invoke them from a perl daemon (amavisd)

    Having both gives us the flexibility that we are fairly pleased with :)

  6. Spot on. Spot on.

    “Democracy In Action — powerful API, alas, not yet well documented.”

    We’re working on it! :-) Look for major developments on this front very soon.

    Let me add two more companies to your list of those that get it: Amazon.com and Ebay. They both do an enormous amount of very profitable business traffic over their APIs every day. Amazon in particular has been very clever about enabling the creation of new types of businesses on their platform; their API is a model, especially from the platform-independence perspective (you can access it via SOAP or REST, and they provide code samples in Perl, PHP, Python, Java, C#, ASP.NET, and XSLT, and those are just the ones I remember off the top of my head.

    We in the advocacy space still have a long way to go before we reach this level of sophistication. Now that APIs have become fashionable, here’s a simple test I like to give people opining about them: ask them to explain briefly to you the key differences between SOAP and REST.

    You’d be surprised how many will tell you that SOAP is something you scrub with :-)

  7. Lobo-

    Thanks for updating that page to reflect your current progress.

    “An open API layer” indeed the goal. And my definition of “open” is that it is open to external applications running on different platforms. My point is that a PHP-only API does not meet that test. If CiviCRM achieves a web services API by building on top of its PHP API, that’s fine.

    Our goal is to advocate for applications that can share data across platforms.

    It sounds like CiviCRM shares this goal. It is good to hear that you have made progress on implementing it.

    Now on to documentation! ;-)

  8. One subtlety here is that merely network enabling native apis doesn’t always do the trick. Web services must expose the proper level of granularity (ussually courser than a native api) for them to be useful. Unfortunately, this isn’t a science yet, and it will take a while to get the hang of this.

  9. It seems like we are committing a whole slew of offenses outlined in Robert Wiener’s 10 common mistakes in selecting donor databases.

    The techies are making the decision.
    Falling in love with cool features
    Buying more than you need
    Confusing Highly Functional Software with Highly Trained Staff

    I think that discussions like this need to be anchored in the experiences of the people we are trying to help.

    Web Services APIs don’t create simplicity, they create power. Power comes at a cost. That cost is things like:

    • Latency. What if your geocoding service is slow, how is the user going to know that the 5 minute import of 10 records isn’t because your software sucks.
    • Breakage. When a web service breaks, the application breaks. And that breakage is entirely out of you our the user’s control. (there are no service level agreements for a collection of web services)
    • Complexity. Web service A updates its API and inadvertently breaks backward compatibility. If your application relies on 10 web services, how do you debug just to figure out what is going on? How do you get it fixed?

    An application like CiviCRM is actually the poster child for exactly what Jon is talking about. We make the web services vision possible with SOAP. At the same time, we don’t think nonprofits should be exposed to the risks of the latest wiz-bang stuff just because it is cool.

    So we are also the poster child for avoiding wiz-bang for wiz-bang’s sake and keeping functionality close to customer needs. We’ve implemented web services for geocoding and mapping from Yahoo and Google. But a user has to make the decision to enable them, which suggests the user might understand the risks and have the technical chops to deal with them.

    We expose our APIs vis SOAP, but we have yet to see someone come along with a critical nonprofit business need to use those SOAP APIs. Are we suppose to invest in things that nonprofits don’t currently need, just because it’s cool?

    I think we are all headed in the right direction and I think we’ll see a great ecosystem develop around nonprofit technology (Drupal, Plone, CiviCRM, CivicSpace, ?). We just need to be careful that real-world nonprofit needs drive the development of that ecosystem.

  10. David-

    Nobody here is talking about selecting donor databases. Nobody’s talking whiz-bang or cool. We’re talking about designing software so that it expands the options available to users and developers, rather trying to force everyone in the sector on a single software stack.

    Unfortunately, in my book, CiviCRM is NOT even close to a poster child for API development. Your rhetoric is way ahead of the CiviCRM code. The biggest gap is that the CiviCRM team hasn’t yet document the web services API in any meaningful way, nor have they built out any web services API calls beyond those that CiviMail requires.

    That renders it effectively impossible for anything that isn’t a PHP app on the same server to access data in CiviCRM. Except for your own CiviMail product.

    If an API gets built but not documented, can it be said to exist? ;-)

    When CiviCRM has API documentation that is even close the the quality of http://www.salesforce.com/developer, then it will be a poster child for API implementation.

    Here’s a critical nonprofit business need for a web services API. Some of them want to use a non-PHP CMS that fits their needs better than Drupal or Mambo, or need to their CMS on a different server from their CiviCRM database. This is not currently possible without a web services API.

  11. Damn, Jon! I was just about to e-mail you to ask a question which you answered in this blog post.

    Well done, and thank you! See you in Seattle…

  12. Breakage. When a web service breaks, the application breaks. And that breakage is entirely out of you our the user’s control. (there are no service level agreements for a collection of web services)

    Don’t conflate building components that expose themselves as webservices with building apps against thrid-party web services. For all the usual f/oss reasons it can be important to own and run your own services. Not just for sla, but for app-specific policy as well.

    Complexity. Web service A updates its API and inadvertently breaks backward compatibility. If your application relies on 10 web services, how do you debug just to figure out what is going on? How do you get it fixed?

    testing testing testing. And decoupled components ought to be easier to test, and potentially more reliable than complex systems. Updates to service apis is no different than the upgrade path on a single platform. Dependency resolution needs to happen in both cases. What is the upgrade experience like in the world of drupal?

  13. Jon-

    You are right. Our documentation is no where near where Salesforce is, nor do I think it will be anytime soon. And we are certainly not the poster child for API documentation. As I have said before, and will say again, a direct comparison between Salesforce and CiviCRM is silly… Salesforce is a much more mature product that does a bunch of things extremely well. CiviCRM is a little over a year old.

    My post was not meant to be rhetoric. As always, we are happy to work with anyone to demonstrate the precise strengths and weaknesses of CiviCRM. There are plenty of both. In fact, the best place to see them all documented in vivid detail is our email list, wiki and bug tracker (www.civicrm.org).

    Here’s a critical nonprofit business need for a web services API. Some of them want to use a non-PHP CMS that fits their needs better than Drupal or Mambo, or need to their CMS on a different server from their CiviCRM database. This is not currently possible without a web services API.

    Absolutely. That is precisely what Salesforce has been architected to do. This is what CiviCRM has been architected to do. I’m sure this debate will die down once a few CiviCRM integrations have been done outside the LAMP stack. As always, were open.

    And for Jonah-
    Upgrades in Drupal-land are a bear. Over the next few releases, the community should get a dependecy system in place that will improve things. We shall see.

    The neat thing about a lot of smart and passionate folks working toword a common goal is that it’s virtually inevitable that the sector is going to benefit. :)

  14. Upgrades in Drupal-land are a bear. Over the next few releases, the community should get a dependecy system in place that will improve things. We shall see.

    I disagree with this :) If you use standard drupal and contributed modules and you don’t fork anything you can simply run the upgrade script between point releases and your set. if you use CivicSpace we have a garunteed upgrade path for users between Dupal point releases. A dependancy system is in the works that will make this process quite a bit easier. The next version of Drupal 4.7 ships with a contrib module upgrade system and an automated module install system. By 4.8 / 50 we should have most or all the pieces in place (hopefully :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Subscribe without commenting