Click. Connect. Learn.

All posts in Drupal

Administering Sites Deployed From Distributions

Within the Drupal community, we continue to wrestle with what, exactly, these things called distributions should be, and could become. Going into D7, the benefits of distributions (and specifically, the tools that support building distributions) are clear to developers. For most shops, the notion of building a site without using (at the very least) features hasn't really been an option for the last few years. Within FunnyMonkey - like at many other shops - we treat custom builds as product development, creating an install profile to contain config that can't be pushed to features, and making use of drush make during the build. Again, this is nothing horribly unique; this workflow simplifies testing, and technical decisions can be directly traced to code or a feature. Over the course of even a mildly complex project, having a clean, replicable build for all team members to work against saves countless hours of time.

Mind the Gap

But this is really only of interest to developers, or to people who directly benefit from a clear development process. These benefits filter down (theoretically) to the end user and content administrator in the form of a site that is easier to manage and use. But, the power of distributions has yet to become obvious to less technical users because, up to this point, the focus of distributions has been on automating a site install that delivers complex functionality out of the box. Distributions have done a great job solving this issue. Between installation profiles, drush make, and features, many of the pain points of replicating site builds have been eliminated or minimized.

So, using distributions, Drupal can be used to create products, and everyone can build an app store, and developers can sit back and make it rain.

Except: while distributions have taken the pain out of replicating complex site builds, they still require ongoing maintenance. Like every other software system, technology requires periodic updates; Drupal is no exception. Any Drupal site, whether built from scratch or deployed from a distribution, requires maintenance over time.

Most distributions are based on a collection of features. As a result, maintaining a distribution raises an additional question: should a distribution attempt to stay synchronized with the features that power the distribution, or with the underlying modules that are used within these features?

Tracking Overridden Features

The reflexive answer - the satisfying answer, the one that we want to be right - is that any site built on a distribution should stay synchronized with the features that drive the distribution. Over time, these features will evolve, and will have new, better, shinier functionality. However, the strength of features makes managing the config covered by that feature potentially more complex over time (there are many strengths and advantages to features, but one of the most obvious is the ability to push complex config to code where it can be managed and tracked via version control). Even a simple change (adding a field to a view, adding a field to a content type, changing a variable that has been strongarmed, etc) results in the feature being overridden. Over time, these overrides will need to be tracked against the features in the base distribution, and reverting these overrides could potentially break customizations in the production site. So, on a production site with overridden features, the overrides need to be tracked and either merged or ignored on each feature update. This is very possible to do, but it requires a level of technical expertise that is generally beyond the immediate reach of the people that distributions are supposed to empower: less technical users who just want to launch and use a web site.

Distribution-based Sites as Standalone Web Apps

And this brings us to the other alternative: once a site has been deployed from a distribution, that site is essentially detached from the distribution, and the site should be viewed as a standalone web app. With this approach, the features used on the production site should be checked into version control, and all site-specific overrides should be committed. This approach cuts the site off from potentially all future improvements in the base distribution, but it ensures that all modifications remain untouched. And, of course, this method requires a level of technical expertise that undercuts one of the main selling points of distributions. Managing overridden features in a sane way requires technical expertise, and distributions are supposed to make it easier for non-technical people to run Drupal sites.

Middle Grounds

There are additional possibilities here for working with and adapting the features within a distribution, but all of these methods require a level of technical expertise that undercuts the "distributions will simplify Drupal for non-technical users" argument. Probably the best option involves having features that are well architected, and make logical separations with documented dependencies within features. Then, end users can turn on the features that they want, and then they can clone and modify or ignore the features that they don't want to use. This leaves the core features untouched and in synch with the base distribution, but it also creates an additional set of features that should be managed via version control.

In any case, no matter the approach, maintaining a complex web site requires technical expertise. Distributions take the pain out of deployment, but they haven't done the same for maintenance.

Where To Go From Here

At the risk of stating the obvious, keeping all features untouched is really only possible in a SaaS environment, or an environment where the site admin has the discipline to resist unnecessary changes to the config. Aside from these two scenarios, most sites will require some modifications to meet the specific needs of the people running the site. And it's worth noting that in a SaaS environment, people are used to not having full admin rights, so there are use cases where only providing limited admin access could work. But, for most people/organizations, one of the key appeals of Drupal is the ability to tinker, modify, and expand the base build. This is good, and normal, and natural; if an organization doesn't think about how to use their web site in different ways after the initial launch, they are essentially stuck in time, and that's not good, and distributions need to be flexible enough to support that reality. However, within the community, until we clarify what we mean when we talk about admininistering a site deployed from a distribution, people using distributions will remain uncertain about the best approach to keep their site working well over time.

Image Credit: "Mind the Gap" taken by ammiiirrrr, published under an Attribution Non-Commercial Share Alike license.

Terrain, and Responsive Design

Like many folks who build web sites, we're seeing increased traffic from people with mobile devices.

In addition to building web sites, we're also geeks, and based on our experience working and playing on the web using mobile devices, we know that people using handhelds want the same information as people browsing the web using a laptop, netbook, or a desktop. Believe it or not, our interests as end users do not shrink or grow in proportion to our screen resolution. As a result, we favor the principles of responsive design as a means to meet the needs of mobile users.

The terrain mole!

We've been using Hexagon as our base theme for the last few years. Hex is insanely flexible, designed to work smoothly with Context, designed to eliminate the pain of building subthemes, and has an architecture that supports functional plugins within the theme. Currently, Hex ships with plugin support for accordion regions, vertical tabs, and horizontal tabs. If you want to get a sense of how Hex works with subthemes, grab a copy of VoiceBox and look at how the Dispatch theme interacts with contexts.

But, because Hex is designed to be extensible (did I mention that Hex is insanely flexible?) it can support other types of plugins.

For example, a Hex subtheme can use a plugin that supports the 960 grid system with a custom fluid width grid. Along with some other modifications, which we'll discuss in another post, this is the basis for our responsive hex starter theme. A development release of Terrain is now available for download on drupal.org. Get it while it's hot!

You can also access this video directly on Vimeo: http://vimeo.com/27829604

Because we believe that one's own dog food is the tastiest, http://funnymonkey.com is now running on a Terrain-based theme.

Code Review: It's A Group Thing

Coming up this Wednesday, May 11th, the Portland Drupal Users Group will be devoting the entire meeting to code reviews of full project applications.

The meetup starts at 6:00 PM, and it will be held at TigerLogic, 1532 SW Morrison St, Portland, on the 2nd floor. I have heard rumors of pizza, and possibly beer. Show up to find out if these rumors are true!

I Hate Wires

During the meeting, there are several things we're looking to accomplish:

  • Make a dent in the full project application queue;
  • Make further inroads into defining a clear, replicable process for code reviews, thereby:
    • Making it easier for new developers to get familiar with Drupal;
    • Making the application review process less painful;
    • Beginning to establish code review sprints as a new community norm, like code sprints and documentation sprints.

If you are an experienced developer in Portland, COME ON DOWN! We need you to help answer higher level questions, and review code!

If you are a developer who wants to get more involved, COME ON DOWN! We need you to ask questions, answer questions, and review some code!

If you are a themer, COME ON DOWN! We need you to help review theme applications!

If you don't write code, and don't theme, COME ON DOWN! We need you to help with documentation, as we will also be working on documentation.

We are also fortunate to have Alan Palazzolo and Chacha Sikes joining us to help run the event.

So, on Wednesday, May 11th, at 6:00 PM come on down, review some code, write some docs, and meet some great folks.

Image Credit: "I Hate Wires" taken by Robert Anthony Provost, published under an Attribution license.

Community Development and Code Reviews

All of what follows in this post is based on two core principles:

  1. Teaching and explaining a topic is the best way to master it; and
  2. The ideals of a community are best handed down from individual to indivdual as they work toward a common goal.

Toward that end, on May 11th, the Portland Drupal Users Group will be using our monthly meeting to run code reviews.

As a side note, while this writeup uses Drupal as an example, the general principles will work across projects, or in training/educational contexts outside software development.

Peer review

Benefits

Code reviews can help grow the existing pool of talented Drupal developers by providing a documented, more structured way for developers new to Drupal to begin learning the Drupal codebase, and learning how to work within the community that maintains with that codebase. Code reviews can also help alleviate one of the more consistent bottlenecks for developers looking to contribute more to Drupal: the project review process.

Code reviews reinforce good habits. The process of reviewing someone else's code forces a few things to happen:

  • The reviewer hones their developer chops by seeing how other people have solved problems;
  • By reviewing code for security issues, developers get more familiar with the best practices for writing secure code, and security issues in exiting code can be addressed;
  • By reviewing code for proper and appropriate use of Drupal's APIs, developers get more familiar with Drupal's APIs, existing modules get more stable and less likely to break on upgrades, and more people become familiar with how core works;
  • By testing and reviewing code via the Coder module, and/or using Devel as needed, easy fixes get implemented more quickly;
  • Opportunities for documentation and better test coverage can be highlighted as part of the review, leading to more automated review and better documentation.

But, most importantly: the process of providing a review of someone else's code creates a dialogue within the community. Code review is contribution and connection, and these are key elements to working successfully in the Drupal community.

Additionally, code reviews provide a clear starting point for people asking how to begin in Drupal. Not many people are going to start reading api.drupal.org without a clear context or reason to do so. Code reviews provide that context, and the process of a code review provides a structure for people coming up to speed on Drupal development. So, when someone says, "I want to get started in Drupal, but I don't know where to be begin," we now have a simple answer:

Check out this page for some background on giving code reviews, and then review some code. The process of reviewing projects will get you up to speed on how to develop in Drupal.

To summarize, code reviews:

  1. Help new developers learn Drupal development best practices;
  2. Help new developers make contributions and connections within the community;
  3. Help reduce the bottleneck of getting new projects approved. And, it's worth noting that increasing the visibility of code reviews as a valued contribution within the community will likely increase the pool of reviewers into the future.

Getting Local

There are a large number of local user groups. While a lot of work can be done virtually, the value of face to face meetups should not be overlooked or underestimated. Local user groups provide an opportunity for more exerienced and less experienced developers to get together and share ideas. If user groups dedicated one to two meetings a year to code reviews - where more experienced and less experienced developers worked together on code reviews - the habit of code reviews could start to become a recognized norm. Once a developer has become proficient in code reviews, the practice of reviewing someone else's code can just as easily be applied to reviewing their own code.

Over time, code reviews should be as familiar and as recognized as Code Sprints and Documentation Sprints. These are held at every DrupalCon, and are regular occurrences at local Drupal Camps. Given the value of code reviews in developing new talent, increasing the consistency and quality of contrib, improving collaboration between developers, and strengthening the contributions of local user groups, code reviews have a role to play in helping the Drupal community continue to grow, evolve, and mature.

Getting Started

Fortunately, getting started is the easy part. The queue for project applications is open, and you can sort by date to see the oldest projects first.

The Code Review Group is getting up to speed, and you can indicate on your profile that you are actively reviewing code. This wiki page and this handbook page both provide instructions for diving in to a code review.

And, if you are in Portland this May 11, come to the Users Group meeting and get running with some reviews.

Image Credit: "Crystal peer-reviewing Mike's code" taken by Jason Crane, published under an Attribution Non-Commercial Share Alike license.

I'd Like A Server With That Web Site, Thank You: Provisioning Servers and Install Profiles

Over the last few weeks, we have done some work building out an automated installation script that runs on Linode. More detail is provided below, but for those of you looking to move on and read something else, here's the short version: it can provision and tune your server, and install your web site, in about 15-20 minutes.

For those unfamiliar with Linode (see note below), they are a hosting company that provides VPS hosting. Within Linode's hosting environment, people can set up stackscripts to automate/streamline setting up a hosting infrastructure.

Atrium

We worked with Dennison Williams to start the process of building this script. It's also worth noting that, while this script is specific to Linode, the steps within this script can be used as the basis of a script to automate an install in a variety of other hosting environments, from Amazon Web Services to the Ubuntu Cloud.

Some of the steps automated by this script include:

  • Provisioning Linux, Apache, PHP, and MySQL on Ubuntu 10.04;
  • Low-level performance tuning for Apache, PHP, and MySQL;
  • General security hardening, including installing fail2ban
  • Installing Drush;
  • Installing APC;
  • Setting up cron jobs;

And, of course, it completes the process by downloading the codebase for VoiceBox, which can then be installed on the freshly provisioned system.

So, in the space of about 15-20 minutes, you can install a completely configured web site on a tuned, secure, scalable web host. Currently, we have this configured to grab and install VoiceBox, but given that this script can be modified to grab other codebases, it can also be used to automate other installation profiles (Managing News, Open Atrium, Open Public, Open Publish, etc) just as easily. This script makes another aspect of getting up and running with installation profiles that much easier.

This also ties in with ongoing work to make installation profiles - and their component features - more reusable. As we learned during our VoiceBox build, creating reusable features for install profiles is easier said than done.

This is where today's news about the acquisition of two of the most popular installation profiles - Managing News and Open Atrium - becomes relevant. Managing News and Open Atrium were built by Development Seed, and were acquired by Phase2; both of these shops have an amazing track record of high quality work, and of contributing code back to the community. This acquisition represents a significant investment in install profiles, which means that the tools to build and maintain them will likely be improving in Drupal 7 and beyond. It also sets up an opportunity to further the conversation about what is needed to take installation profiles to the next level. Given that we have been building and distributing pre-configured sites since 2005, we're looking forward to continuing our work in this space.

So, as we continue to look at the process of building and maintaining web sites, using installation profiles takes the sting out of the initial site build. Reusable features have the potential to allow useful functionality to be built once, and reused in multiple places.

Automated provisioning scripts take the sting out of building a secure server infrastructure. For many sites, using the advantages of cloud-based hosting takes the sting out of scaling a site.

To be clear, both on their own and collectively, none of these approaches are silver bullets, and running a moderately trafficked web site still requires maintenance time and technical expertise. But, these approaches make running a successful web site more accessible to more people, and will allow greater numbers of people and organizations to get more done with less work and less cost.

Image Credit: "Atrium" taken by Schrottie, published under an Attribution-NonCommercial-ShareAlike license.

NOTE: We do not have any type of sales/affiliate relationship with Linode (or any other hosting company); in our experience, affiliate agreements destroy the credibility and therefore the usefulness of any recommendation. We are, however, using Linode for several internal projects. If/when we discover other options that offer similar value, we'll definitely talk about them, and probably start using them.

Open Content, Where You Want It, In A Space You Control

In the past, we have talked a fair amount about open content. Done correctly, open content has the potential to support meaningful change within education. A significant amount of open content exists on the web; much of this content, however, is not immediately reusable.

In this post and the accompanying screencast, we break down a system that can be used for authoring, redistributing, remixing, and collaborating on open content. The site is customized on top of VoiceBox, and all of the components used to build it are freely available. I'll be presenting on this topic in more detail at Educon in a couple weeks.

This system has several differences from other systems that support open content, but one major difference merits additional notice: it is designed to be porous, and it is designed to be installed and run by the people or organization creating the content. It can be as public or as private as you want it to be, but that decision is left up to individual content authors, as opposed to the people building the content silo.

Overview

Within the site, people can:

  • Create and join groups; groups are workspaces that can be fully or partially private.
  • Author notes; notes are the main form of content within the site. A person creating a lesson would build it by creating notes.
  • Start discussions; discussions occur within groups, and provide a means to discuss areas of general interest, or to discuss a specific note.
  • Offer feedback; feedback is private between the person receiving and the person offering feedback.

Notes can be created as individual pages, or organized hierarchically into multiple page books. Each book can be accessed in print-friendly format, and each book generates its own RSS feed that can be used to import the book outside the site.

In conjunction with a service like BookBrewer, this site can be used by schools or other organizations to create, distribute, and sell eBooks. Also, given that this site is built on Drupal, we just might write some code that allows Views to output content in ePub format.

Video Breakdown, Education Version

Use the minute marks to skip around the video to find the sections that interest you most.

Creating Notes | 0:15

The opening section breaks down how to create notes, and how to shift pages within a book. Notes can contain a variety of different media, and in this section we talk briefly about embedding images, audio, video, and other files within your notes.

Groups | 4:15

This section describes how to use groups to work collaboratively. It shows how to track activity that has occurred within groups, provides an overview of group home pages, and shows the different ways that users can choose to share their content.

To see an example of how to create a group, skip ahead to the 15:00 mark of the video.

Discussions | 7:10

This section shows how to use discussions to collaborate within the site. Discussions can be tied to a specific note, or more general within groups.

Feedback | 9:30

Feedback is a private communication between a reviewer and an author. For example, a teacher can provide private feedback to a student, or a department head can provide feedback and evaluation to an instructor in their department.

Moving and Copying Content | 12:30

In this section, we show how people can work with other people within the site, and how work can be distributed outside of the site. We highlight how to access the RSS feed, and print-friendly format of each book.

For people working in the site, we cover how to copy books between groups to allow people to derive copies and remix content without disturbing the work of their peers

Video Breakdown, Drupal Version

This breakdown gives a high-level overview of how the functionality was created. As mentioned earlier, the site build is based on VoiceBox. The main additions within the Open Content site include the ability to create multiple page books (leveraging core Book module and the contrib Book Manager) and the ability to form working groups (using Organic Groups).

Creating Notes | 0:15

Notes are created using CCK, and the Filefield, Imagefield, and Embedded Media Field modules. With the exception of the embedded media field, this setup is identical to what we are using on VoiceBox. Audio and video nodes are formatted using SWF Tools, and images are displayed using LightBox2. We will likely be switching to ColorBox in Drupal 7.

Groups | 4:15

The groups functionality is created using Organic Groups. The additional access control options for groups are created using the OG Access Roles module.

Discussions | 7:10

Discussions are structurally identical to Notes, with the exception of a nodereference that allows a discussion to refer back to a specific note. The nodereference field is populated using the Node Reference URL Widget module, and the link to create the actual discussion is created via Views.

Feedback | 9:30

Like discussions, feedback nodes use a nodereference to point back to a specific note. Additionally, feedback nodes make use of the Node Access User Reference module to control access to the feedback node.

Moving and Copying Content | 12:30

The ability to copy books is done via the Book Copy module.

Additionally, when books are placed into a group, the group affiliation and the level of access control is kept in synch throughout the book tree via the Book Manager module.

The RSS feeds for books are generated via the Views module, and the print-friendly format comes from the core Book module.

Conclusion

If there any follow up questions, please feel free to ask them in the comments.

Community Media and Grassroots Outreach With VoiceBox

Many moons ago, we announced that our proposal for the Knight Drupal Initiative had been funded.

Yesterday, we pulled back the curtain: the alpha release of VoiceBox is now ready for download.

The site was designed to support community and grassroots media organizations. Out of the box, it supports the following features:

VoiceBox logo
  • Customizable user profiles;
  • Discussions with embedded images, audio, and video;
  • Publishing pictures, audio, and video via mobile devices and/or email;
  • Content aggregation via rss;
  • Remixing, curating, and republishing content within issues;
  • Sharing content to external sites or via email; and
  • Slideshows of featured content on the site homepage and selected landing pages.

As site members create content, they generate their own blog, and each member has a personalized dashboard that gives them an overview of the site.

While the site was designed to support community media and other grassroots organizations, the base functionality can be used in a variety of different contexts. It could easily support a school newspaper, or, with some modifications, it could work as the public-facing site of many organizations that wanted to communicate with their stakeholders and develop a community around their work. In particular, the blend of content aggregation, internal site discussions, repurposing content within issues, and sharing out to external sites via social media helps connect the smaller communities of individual organizations with larger external groups that share a common interest.

The installation profile installs like a regular Drupal install; for people with experience working with Drupal or other open source systems, you can be up and running in around 10 minutes (This video covers the initial installation and some post-installation customizations; it was shot in real time, and it runs about ten minutes).

Additionally, the site is documented, and the documentation includes a set of screencasts.

At the risk of stating the obvious:

The development of this project would not have been possible without the generous funding of the John S. and James L. Knight Foundation. Without their grant support, we would not have been able to devote the necessary development hours required to complete this project.

It's also worth noting that this project could not exist without the countless person-hours that have gone into building Drupal core, and the various contributed modules. In particular, the work of Development Seed deserves special mention. An incomplete list of Dev Seed-powered functionality includes Feeds, Context, and Features; this foundation made much of we built possible.

So, what are you waiting for? Grab the code and get started!

Features, Dependencies, and the UI

A few weeks back, we wrote about our thought process for building features, and how we wanted to make sure that the features achieved two goals:

  • Be part of a fully functioning install profile that "just worked"
  • Be as reusable as possible within VoiceBox variants, or within other sites.

As part of achieving these goals, we created an option within our install profile that allows a user to select the Full or the Minimal install. The full install runs through the complete install, and creates a fully functioning site.

Minimal or Full install option

The minimal install, however, runs through the install profile, and then allows the site builder to enable the selected features they want.

Features are modules; this allows us to set dependencies between features. However, as we built out features based on our site configuration, we realized that the User Interface elements created the largest set of dependencies - and this makes sense, because a good UI collects up the various sets of functionality and presents them to users in an intuitive, coherent way. To account for this, we split out our UI into separate features, so someone doing a minimal install can choose to use these defaults, or build their own UI.

A graph of the dependencies shows this better than words can explain it:

  • Dependency tree of VoiceBox without any UI (click the image to see a full-size version):

    Feature dependencies with no UI

  • Dependency tree of VoiceBox with the UI features included (click the image to see a full-size version):

    Features dependencies, including the UI

    The elements pictured in blue are the UI-related features; within the site, they expose functionality to end users - this is also known as "making things useful or usable."

    Within the features, UI elements create a higher level of interdependence.

We are running through the last few rounds of testing here, and barring any nasty last-minute surprises, we should have a release of VoiceBox by tomorrow. Our features server, which will be launched with the site, will allow people to get the entire install as a package, or the bundled features individually.

An Incomplete History of Sexism In Drupal

So here's the thing. I'm proud to be a member of this community. The Drupal community contains some of the smartest, kindest, most generous people I have ever had the privilege of meeting. I don't think I'm overstating anything to say that my involvement in the Drupal community has altered the trajectory of my life. I have learned more in my six years of work within and around this community than at any other period of my life. It's an amazing place.

And that is why I'd like to see us do better. We can always rationalize away the things we don't like, or find ways to justify things that are distasteful.

But if you see something that feels wrong, stop and ask questions. Realize that you will need to have the same conversation, repeatedly. Realize that people will get mad at you and blame you for bringing it up. Realize that in the process of having conversations, you will learn about things that you misunderstand as well. But don't stop having the conversation, because that's how change occurs: one awkward, uncomfortable, unwanted conversation at a time.

Syndicate content