Drupal

Whitehouse.gov is now live running Drupal. It's great to see yet another example high-profile Drupal site go live. I'm curious to see how this story unfolds over the next few days.

I haven't spent much time going through the site, but I'm glad to see their implementation of Solr for the search features. Transparency comes easier when you can find what you are looking for.

I was also glad to see that they have implemented RDFa. I'd love to see similar steps taken on other government sites to increase transparency and interoperability.

The Washington Post has an article about the launch. It's worth reading for no other reason than to see how it sounds when a non-technical person attempts to cover technical issues. For what it's worth, this would have been a great time to use link journalism -- they could have collected up links to people who actually understood the significance of the story, and who were capable of articulating the narrative.

Update: The Personal Democracy Forum has a great post on this. Among other details, it credits Acquia and Phase2 with working on the site. Congratulations to all involved.

Over the last weekend, we had the opportunity to install and test Managing News, a Knight Foundation funded project built by Development Seed. Managing News will be released later this week; we were fortunate enough to get an early preview.

Brief Overview

Managing News is an install profile built on Drupal. All of the components used in Managing News are available for free under open source licenses.

In short: it is free to obtain, and it installs like any other Drupal site. For the visual and auditory learners, this video -- produced by Development Seed -- provides an overview of the site.

Managing News contains three sections: Feeds, Search, and Channels.

Each of these sections is covered in more detail below.

Digging In: Adding Feeds

At a first glance, some people might confuse Managing News for a feed reader. This misconception is understandable, as the first step in using Managing News involves adding feeds to bring in content. This process is straightforward: click the "Add Feed" link, as shown below.

Once feeds have been added, information from these feeds will be imported into the site. As data begins flowing through the site, more of the features of Managing News can be used. Every section of the site (Feeds, Search, and Channels) contains consistent display options:

1. Title, with summary, organized chronologically;
2. List format, 1-2 line summary, organized chronologically;
3. Visual representation on a map.

These options allow people to navigate the news in the way that best suits their need. Additionally, as people read through stories, they can share posts that they see by clicking the "Share" icon, as shown below.

When content is shared, the path to the original article is included as the link, and the url is automatically shortened. The included link points back to the original place where the article was posted, so the original source receives credit -- and the resulting web traffic -- for their post.

The site streamlines the process of sharing content to Twitter, Facebook, and via email.

Two Notes Before Moving On

Two additional notes before we move on:

Note 1: this post covers one one way of bringing data into the site -- via RSS feeds. However, content can be brought into the site in a range of other ways, including via CSV import. Some of these additional options are covered later in this post.

Note 2: mapping works out of the box. As posts come into the site, geographic information is automatically extracted. This allows posts to be displayed against a map to highlight relevance to a specific region. To emphasize: this mapping functionality just works, with no additional configuration required. Moreover, it has been designed to be customized and extended as needed, but more on that later.

Search

Once feeds have been brought into the site, information can be sorted and discovered via text-based search. Searches can be saved; this way, if there is a specific type of information that needs to be highlighted or discovered from the incoming information, the saved search can help make this happen automatically.

Saved searches also generate RSS feeds, so people can subscribe to these results.

Channels

Channels provide an additional way to vet, display, and redistribute content. Channels can be created by site members; once a channel has been created, people can tag individual articles to be published in a channel. Like saved searches, channels generate an RSS feed, so people can subscribe to a channel.

Channels are created from the Channels page, or when viewing the search results.

To add a post into a channel, select the active channel, and then click the icon next to the post.

Taking a Step Back

The natural flow of information within the site -- from all feeds, to saved searches within these feeds, to channels that group and recontextualize individual items according to an arbitrary theme -- helps illustrate how the site can be used in different ways by different people within the same organization. The Feeds page functions much like the home page of a newspaper, magazine, or blog: it shows you all of the latest news. For people who are more focused in what they are looking for, the Search page allows them to carve through the content by searches. Finally, for people looking to browse through content on limited time, the Channels provide information that has been vetted/singled out as having greater value.

And this is where the real value of Managing News begins to become clearer: with most products, you can break down the value of the product in an answer to one simple question: what does it do? With Managing News, the breakdown is not as simple, as it does different things for different people at different times. Moreover, these general categories (feeds, searches, and channels) can be used for different things; as one of many possible examples, a local paper could use a channel within Managing News as a tipline; people could email tips into the paper, they could be imported into Managing News via Mailhandler, and the more promising leads could be highlighted in the Tipline channel. A similar process could be used to sort through hashtag-based coverage of breaking stories via Twitter or other social media channels: posts with the hashtag could be imported, and then a selected number of these posts can be republished in a channel -- and, as we discussed earlier, the channel would have its own RSS feed, making the channel a cleaner version with a better signal to noise ratio than the original disparate sources.

A Product and a Platform

In its current form, Managing News provides powerful functionality. The standalone product will allow many organizations to extend their online presence with little to no additional expense. This is a tool that levels the playing field by giving smaller organizations access to tools previously reserved for bigger, richer organizations -- however, it will likely be adopted and extended by various types of organizations because it is both easy to install and easy to extend.

It's very easy to see how an application like EveryBlock could be built on top of Managing News -- with the caveat that Managing News could be developed to simultaneously support the hyperlocal, the regional, and the national. Using building permits as an example of just one of the nearly countless potential data points, a site like Managing News could collect building permit info for any city that made that info publicly available in a readable format. Then, that information could be displayed on a block by block basis within a city (like Everyblock currently does) or it could be used as the basis for comparing building activity across regions, across time, or against other data points that have been imported into the site. The geotagging would need to be modified from the default configuration, but the system has been build to support these types of customizations.

Managing News could also be used to give organizations an internal version of something like Publish2. Where it really starts to get fun, though, is that Managing News-based services wouldn't need to compete with an application like Publish2, they could actually work alongside it in a mutually supportive way. An organization could have their internal system based on Managing News, and then create a publicly accessible channel that would connect up to Publish2 by extending the "Share" feature described earlier in this post.

For our part, Managing News provides us some great opportunities for our own Knight-funded work. The aggregation-collection-republishing workflows can be leveraged as part of our platform, and the fact that Managing News exists allows us to focus in on other aspects of development, such as harvesting data from handheld devices. This collaboration highlights another advantage to developing these tools within an open source ecosystem: in the process of doing our work, we will contribute both code and documentation back into Managing News. The existence of Managing News will improve the quality of our work, and in turn, our work will filter back into Managing News.

Conclusions

Managing News gives organizations a powerful, flexible tool to use as they work online. The functionality of the site is well defined, and cleanly focused. Moreover, the design of the site keeps things looking simple, when there is some fairly complex data management occurring. The official release will be announced later this week; watch the Development Seed blog for the announcement.

I'm getting ready for the Future of News and Civic Media Conference, and as part of the preparation we have been putting together a research/development site as part of our work for our KDI project. We are still evaluating the different options that will make it into the initial versions of our platforms.

For this stage of the research, we chose to focus in on the events of the Iranian election -- first, I was woefully underinformed about the events of this election, and given the noise (or, some would say the lack of it) about the event, this seemed like a good opportunity to set up a tool that would provide an overview of the event, with a cross-section of primary source material (largely from YouTube and Twitter) and content from more polished sources (both blogs and traditional/mainstream media). In a few ways, this provides a real use case: if an organization doing grassroots organizing wants to find out about and publicize events occurring in several places on the same day, this type of aggregation from multiple sources allows real-time data collection from disparate sources. People can continue to use the tools they already use to discuss their work, and the main site can collect information from these sources and present/recontextualize it from a central location.

With minimal effort, we were able to put together a rough set of tools that allows people to get a perspective on the events going on in Iran. We built the site using tools freely available within the Drupal community. The bulk of the heavy lifting is done using FeedAPI and friends, and the folks over at Development Seed deserve huge kudos for unleashing these tools into the world. We also used the Views module to split out the content of the feeds. Obviously, the resulting site is a proof of concept, more of a pre-alpha prototype than anything else, but the site is useful as a research tool. We'll also continue developing on the site after publishing this post, so the site will undergo changes over the next few days as we modify things/tinker.

On our testing site, the Twitter traffic provides a pretty scattered overview, but taken in aggregate it allows one to scan raw data over time and get a sense of the ebb and flow of events on the ground. Initially, this feed was pretty free from spammers, but lately some opportunists have taken to using the #iranelection hashtag to get a broader audience for their content.

The YouTube videos provide another means of getting a sense of what is happening. As with any piece of media, the source of the content and bias of the author need to be taken into account. Also, our means of collecting these videos is certain to miss some content, as we are just aggregating the feed for the search term "Iran election protest".

The content coming from MSM outlets and blogs is admittedly arbitrary -- we tended to favor sources that had a clean tag for either Iran or the Middle East; the dearth of effective tagging on information coming from both traditional news outlets and some group blogs is discussed in more detail later in this post.

The important piece of this from our perspective: this tool can be easily built and focused on just about any topic. When it is set in motion, the site will gather and import information that can be used, analysed, recontextualized, or otherwise modified. This can be a tool used for any topic discussed on the web:

• grassroots organizing/community media -- use various data streams to collect information in real time that can be analyzed/collected/synthesized over time
• lesson creation -- aggregate writing about a specific topic, then choose the imported resources that align with your learning goals. Edit these assets as needed, or add in information that is missing
• farmer's markets -- farmers/sellers/market organizers use a microblogging platform to describe what they will be selling, and where; this information can be aggregated and geotagged, allowing an accurate breakdown of what is for sale at local markets.

As we built this out, we encountered some surprises. A short list includes:

The Wall Street Journal uses feedburner as their for their RSS tracking. However, this is exposed in their feeds (or at least in their World News RSS feed), and the original URL of their article points to http://feedproxy.google.com, as opposed to a location within WSJ.com. Additionally, the only tag for all content coming out of this feed is "Free". At the risk of stating the obvious, tagging all posts in your outgoing RSS feed as "free" is worse than useless. I have a hard time believing that they don't have the resources to do this well, which makes me wonder why it is allowed to be so sloppy.

NOTE: The following paragraph was edited because, well, it is completely wrong. The Huffington Post nails syndication. The links on the syndication page point out to RSS feeds of several dozen tags. In short, it rocks.END NOTE

The Huffington Post, which makes extensive use of tags to categorize posts, only offers 4 RSS feeds (Full feed, Latest news, The Blog, Featured posts) of content. Even though you can browse posts by tag on their site, you can't actually aggregate by these same tags. Given how easy it is to expose the content of any tag via an RSS feed, I can only conclude that the choice to not support feeds based on tags is tied to their business strategy. Given how little of the Huffington Post homepage is actually original content, it's surprising to see them reducing the number of ways people can interact with their site.

As a final note on this, it's not uncommon to see other more popular group blogs/major news outlets doing similar things. Talking Points Memo eschews aggregation by tags, and none of the major papers do much beyond feeds that summarize articles appearing in their standard sections. Within their RSS feeds, most major papers do little in the way of tagging content. Additionally, most papers/blogs include little more than a brief teaser within their feed. Given that most of these sites devote a fair amount of screen real estate to advertising (and some, like the NY Times, even embed ads in their feeds), their desire to bring eyeballs back to their sites is understandable.

However, an advertising-driven paradigm seems unlikely to work, and it seems especially shortsighted given that excessive reliance on advertising money is frequently cited as a contributing factor in the decline of newspapers. The new media economy seems unlikely to be a link economy; micropayments, paywalls, and/or "better" targeted ads feel equally fruitless. A remix-with-attribution economy feels more likely, with the looming caveat that no one has really figured out how to make that work in a way that makes all the people in the supply chain happy. But the necessary first step is to move away from the notion that the finished work is the starting point or ending point for profiting from that work; that places too high a value on the role of content, and how people interact with information. Content -- the article -- is the middle point of the process, and on the web "content" can be understood as one point in an ongoing chain of synthesis/recontextualization.

Working with aggregation -- arguably the simplest means of republishing and recontextualizing content -- gives an incomplete yet suggestive view into two elements: how an organization understands the web, and how they view the role of content. From what I have seen, both mainstream news outlets and popular blogs do a poor job of making the most of their content. If I had to reduce this down to a single reason, I would say that there is a perceived need to control how people consume content, and that this is tied to the need to pass eyeballs over ads.

However, tethering the distribution mechanism of online content to a strategy designed to generate more pageviews (ie, News as SEO) seems destined to fail, as the gimmickry of SEO doesn't mix well with unbiased reporting.

And I would love to end this post with the next great idea on how to support working writers within this model, but hey, it's late and I need to pack. But I'm looking to forward to learning more about other approaches over the next few days. I'll write up any ideas as they come, and for those who want to follow along, dip into the feed.

Part of our work for the Knight Drupal Initiative includes work to lower the barriers to getting started using Drupal sites to power community-driven media. As we work toward this end, we have multiple tools at our disposal, including:

• Better documentation, for both site administrators and people using the site
• Easier installs, to allow people to enable more complex functionality with less work
• Easier and more flexible theming

At the risk of stating the obvious, the work we do for the Knight Drupal Initiative will also be immediately transferrable to all of Drupal development.

Getting Started: Journal

The Journal module allows us to keep a running record of changes made on a Drupal site. Currently, it allows you to take notes as you build out your site. It's a great tool to track the specific steps people take as they work within a site.

We will build out two additions to Journal:

1. Tag journal entries, which would allow us to create categories of journal entries. For example, while configuring the WYSIWYG API to work with Filefield and Filefield Insert, we could tag these entries as "Text Editor" -- later, this would allow us to see exactly how the editor was configured.

2. Export journal entries by tag or by date range, in csv, ordered list, or unordered list format. This would allow us to generate documentation on how sites are built. This documentation can be used on drupal.org, for site administrators as sites are delivered, or to create end user documentation.

By way of example, let's say we have built a site that is designed allow teachers to create online portfolios as part of professional development. Prior to delivering the site, we could walk through the steps of creating a portfolio. At each step, we would comment on the process of creating the portfolio using the Journal module. We would tag each entry as "portfolio creation." Then, once all the steps have been completed, we would export the steps as an ordered list, and voila, we have our end user documentation.

Moreover, this process also gives us the instructions and order we need to develop functional tests using Simpletest.

With these additions to the Journal module, we will have the base package for creating more comprehensive documentation about all aspects of building and using a site. However, one of the challenges of documenting a site build is getting the documentation into the hands of end users, or people not directly involved in the build. Often, comprehensive documentation can be overwhelming as it floods someone with information.

Inject Me!

The Advanced Help Injection module builds off the Advanced Help module, and allows us to draft documentation using Drupal's core book module. Then, the documentation can be inserted on specific pages within a site, so people only see the documentation they need when they are at a place where they need it. So, the documentation generated using the Journal module can be presented to people exactly when it makes the most sense.

And finally: Advanced Help Injection allows the documentation to be exported as a module. So, all the documentation can be pushed to code, and delivered with the site. This brings us closer to the goal of having a site that is self-documenting, for both site admins and end users.

Moreover, the process of creating these tools can actually be documented using the tools themselves. While it feels like a blend of eating our own dogfood and chasing our own tail, it works.

What Are We Documenting, Anyways?

In this post so far, we have looked at simplifying the task of creating documentation, and using existing tools to get that documentation to people in a more efficient way. However, documentation assumes featuresets, as we can't have documentation about what to do without something to do.

This is where we cue the Features module. Take a minute to check out the latest post illustrating the power of Features over on the Development Seed blog.

As the video and post shows, the Features module allows us to create bundles of featuresets that can be moved from site to site. We envision a system where two things happen:

• Featuresets are paired with documentation, so that any set of features can be accompanied by inline documentation; and
• The process of creating new sets of features is fully documented using the Journal module, to empower more people to build and share features.

Making It Look Pretty

Theming and design are both enormous categories, so I'll be brief. In parallel with the work described here, we are working with Joon Park to build out a base theme (currently named Annex, but it might get changed to Mango Smoothie for the official release) that simplifies the process of creating a unique look and feel for your site. For end users, it supports some simple eye candy out of the box; as one example, it has multiple block regions, including some that support accordion and tab effects by default. For themers and designers, the process of inheriting features from the base theme to subthemes has been overhauled and simplified, allowing people more control and flexibility in building sub themes. More to come on this; ideally, we will have a release at some point in June.

Earlier this spring, the Knight Foundation let us know that our proposal to the Knight Drupal Initiative was accepted.

We are equal parts honored and excited, and work is underway.

Our project targets community media, and seeks to lower the barrier to entry for communities looking to collaborate with other like-minded groups via the web. One of the uses for our work will be within journalism, but other uses include collaborative creation of open courseware.

To be clear: this project will simplify the process of creating, distributing, and using open courseware. We want schools to spend less money on textbooks, and our work -- which will be freely available to all -- will support that goal.

As the project progresses, we will update this space with tutorials, status updates, etc. This page collects our Knight Drupal posts, or you can also subscribe via the feed.