(note: this article was previously published in French in December 2012 - if you have any feedback on the translation made by Coralie Mercier, Goulven Champenois and Stéphane Deschamps, please let us know)

The need

Let’s assume that you browse some websites every day. To do this, you typically use a Web browser such as Internet Explorer, Firefox, Chrome or Safari. When you first visit a site, you discover contents and services. Fantastic. Now, fast forward several days and let’s imagine that you feel the need to return to the website. One frequent question is "What’s new?". You first move will be to look for a "what’s new" section, or you will search directly for new contents, new products, new news, etc.

Imagine now if you had to do this for several websites, for instance to:

• Monitor new ads that are posted on several sites for a given product,
• follow news published in various media outlets,
• pick up the latest posts of one or more blogs,
• monitor the activity of one or more competitors,
• keep up to date on resources and magazines, professional or not,
• aggregate on your desktop,
• …

How to carry out these tasks?

The first solution is to go to all these sites, and for each of them, see if something new has been published. Even if the updates are easy to find, you may have to spend a lot of time, and particularly, during all this time you will need to be connected.

Otherwise you need as many open tabs in your browser as you have resources to look up while you are offline. There are better ways however, one of which is of particular interest: Web content syndication.

What is it?

For content producers, Web content syndication consists in providing a feed of information that contains all or part of available contents. In practice, a file (in either RSS or Atom format) will be updated on the Web server every time a new content is published. This file is available on the Web server and is readable from the outside. Having a defined structure, it is machine readable.

This isn’t a new technology and Openweb had already covered the basics 8 years ago (in French):

• Introduction to content syndication with RSS (in French)
• Building an RSS 1.0 (in French)
• Determing the update frequency of an RSS format feed RSS (in French)

Note: according to Wikipedia in French, Web content syndication is a form of syndication in which a portion of a site is accessible from other sites. This additionally requires granting a license to users of this content. Syndication usually means providing an RSS feed containing the first few lines or paragraphs of the latest items added to the website —this could be a piece of news, or the latest forum post.

Well, that’s perhaps a bit abstract, it is time to look at a real syndication feed.

1. In Firefox, select the View menu> Toolbars menu> Customize
2. In the list of icons that appear, choose the "Subscribe" icon and place it next to your address bar.
3. From now on, this button will be enabled every time your browser detects one or more feeds in the current page.
4. If you’re not there already, go to the address http://openweb.eu.org and you will see that this is the case, thanks to the code below:

<link rel="alternate" type="application/rss+xml" title="Syndiquer tout le site" href="http://openweb.eu.org/?page=backend" />

Let’s have a look at a syndication file, here is for example the main syndication feed of Openweb (http://openweb.eu.org/?page=backend).

This file contains items corresponding to the latest articles. You can look at the source code if you’re curious.

How to read the file?

At any time, you can access this file from your computer and display the information it contains.

Here is how it looks in Thunderbird, for example:

Let’s now consider how and where we can display this content. There are several options:

• View syndication feeds in a software program installed on your computer called News aggregator. It can:
  • be dedicated specifically to reading and managing feeds (e.g. Vienna),
  • be dedicated to other tasks (email software or web browser, which will serve as aggregator).
• View a syndication feed using a Web service (Google Reader, Netvibes, see Web-based News aggregator)
• View a syndication feed in a Web page or Website (look at the aggregation feed proposed by Openweb on its home page).
• View a syndication feed on a refrigerator. And why not, it may become possible?

Practical example: Imagine now that you want to read news published on several media outlets. You will need to subscribe to the syndication feeds made available on each site, but right after that you will be able to see all new information conveniently gathered in one place. Within seconds you have gathered an overview of new content published on multiple Websites. The available content usually contains a title, an abstract, and sometimes the full content. If this isn’t the case, you have to follow a link to the Website in order to access the content in its entirety. Of course, everything that’s already been downloaded on your computer remains available offline, without internet access. From the point of view of productivity or watch, it is a formidable technology. An operation that used to require more than a few minutes and lots of clicking can now be done in mere seconds.

This is not the only advantage: syndication technology is inherently accessible. It is so in the broad sense that this content is easily accessible to all, but also in the sense of accessibility for people with disabilities, since this operation requires providing content in a machine-readable structured format.

The technology is available, relies on open formats, is widespread, relatively easy to deploy, and yet it is endangered. Here is why.

The Dark Side

Syndication seemed a promising technology and yet it has not developed as strongly as might have been expected. Who is there to blame? Well, quite many people: everyone of us.

Content producers

Put yourself in the shoes of a content producer: a user with an aggregator will not always get the news from the site. The user will first look at the title, read the excerpt and then perhaps come to the site. If the content of the article is fully included in the feed, the user will not necessarily have a reason to visit the site. This isn’t neutral in terms of audience, advertising, number of page views. As you know, it is often costly to publish quality content on the web, and profitability is a key issue.

It is therefore necessary to measure the off-site views audience. Some services exist, but somehow, there is a real loss of control, a real let go for the creators of sites to offer syndication feeds.

People often dislike letting go of the way their contents are presented, but it is not all bad: some visitors prefer viewing syndicated content that allow for more freedom in consultation, and especially an easier adaptation to their needs. In addition, consistent display of the stream in an aggregator fosters productivity gains. In this case, offering syndicated content can expand one’s audience.

Finally, producers of syndication feeds, even if they let go of certain aspects, are and remain owners of the feed, which is less clear on platforms such as Facebook or Twitter, which tend to replace them as we will see later.

Browsers

In their eagerness to simplify their interface, browsers have removed the button that signals the presence of a feed and allows subscribing to them. You should know that for browser vendors using pixel real estate for functionality has a cost, and implies an additional mental load for the users. In reality, each button proposed as part of the browser intallation has an impact on the user learning its functionality. In the case of an advanced functionality such as syndication, leaving the button available by default would have represented a militant gesture for the open Web but for several reasons including cost, this is not the choice that was made.

Chrome got the ball rolling, followed by Firefox, and this is probably one of the most pernicious errors for the open Web in recent years. As you may have seen, there are ways to restore the buttons, but they require action from the user.

Users

Yes, users do not necessarily want to subscribe to this technology. The approach of downloading information available in feeds is not so obvious. Referring to one of RSS feeds of the newspaper Le Monde is not the same as visiting daily the website of Le Monde. This is probably the reason why the technology is usually used by the most informed and geeky users. It is probably necessary to make it mainstream, we will see later on how to do this intelligently.

Third-party services

Third-party services such as Facebook and Twitter are increasingly often replacing aggregators in Internet usage. These tools bring together in one place information from many sources. They are not without problems though, from proprietary formats to APIs that can be shut down at any time. This is what has recently happened when Twitter decided to close the doors of the ecosystem that helped build it.

Syndication is perhaps not dead

A Technology for users

If you love the Web, you can only love syndication. Here is a technology that allows users to control how they access Web content and provides access to information quickly and efficiently. It is also a technology that makes content accessible in the broad sense and that allows easy sharing and referencing. By allowing users to perform within seconds tasks that would take hours (check for new content on 50 sites for example), syndication really is a tool that improves productivity; it is also a fantastic tool for tech watch.

Syndication is a key component of an open, efficient and sustainable Web. This is a technology that is made and thought for users; for a quality controller, a technology that meets the requirements of users is necessarily a competitive advantage for content producers who decide to genuinely adopt it.

Another mind shift

However, content producers must learn to relinquish some of the control they had over the dissemination of their information. In this sense, syndication is one of the many artifacts of the reversal of the paradigm imposed by the web: the producers put out an offer, users choose whether to have it. If your content is bad, nobody will subscribe to your syndication feed, however, if thousands of people subscribe, you will find a way to promote your content, you’ll be spoiled for choice.

A real management tool

We talked about articles and news, but if something new appears somewhere and it is possible to turn this into a feed, why not make it possible to subscribe to the latest training, the latest hires in your company, the latest bugs fixed in your application, the latest references to your company, or the latest jobs for your clients? Feeds are an effective way to display your news where there might be added value: your professional networks, on other services related to your business, if you upload presentations, videos, etc. The only limit is that of your imagination.

For example, a beneficial side effect of syndication is to relieve the load on the mail client: instead of having to wade through hundreds of e-mails, which is extremely time-consuming, feeds have many advantages: not only are they a reliable unique source (contrary to systems where forgetting to cc someone can have a negative impact), but also feeds are much lighter than traditional e-mail. Savings in storage and more generally in download time can be significant.

A tool for communication and Web quality

But there is more! Feeds contribute to the selection and dissemination of content on your sites. So when you manage your site yourself with a CMS, there are many modules that allow you to get exactly what you want and display it as part of your site. It can be the feed of an author, a topic, or a specific tag. You can retrieve feeds and update your site on a given topic (e.g. Google News or your Seenthis) making its content more relevant and lively. Through the same mechanism, your own syndication feed is available to be displayed in other sites, and as such generates inbound links, traffic and SEO.

The fight goes on

There’s simply no questioning the usefulness and convenience of syndication. However, it is not as widespread as it should. In my Web quality course if I find that my students do not know this technology and do not use it daily, I pause the lecture, turn on my aggregator and demo how I refresh the 50 feeds that I watch daily, and I tell my students that failing to use such a technology for tech watch is not an option.

A Web professional knows how to work with syndication. Better still, a Web professional should not work without.

It’s probably time to explain again the usefulness of syndication. Let’s fight for default syndication buttons in our browsers. Let’s publish articles, let’s encourage our users to subscribe to our feeds, let’s put the syndication logo back in fashion. In short, it is high time we fought lest this technology falls into oblivion in favour of feeds provided by companies that may shut them down at any time. At a time when Twitter APIs are closing, know that what I am describing is a reality. This is not speculation: companies such as Twitter increasingly restrict access to their API.

For non-professional users, it may be necessary for syndication to happen without their knowing. This is how it is done in several mobile applications, where users consume syndication without ever realising it. We would then have a Web made of invisible feeds. After all, why not?