Superfeedr Blog : Real-time cloudy thoughts from a super-hero

Protocols over APIs

2012-05-15T00:00:00-07:00

There is not a week where I don’t see developers discovering that a given API has been deprecated. Today’s disappointment (at least in my twitter stream) comes from the Google Chart Tool API.

Don’t get me wrong. I feel for these developers and I would be upset should the same problem happen to me. Now, being upset is great. Learning how to prevent that is better, and I believe there are many ways to avoid that. The most obvious one is to re-invent the wheel as suggested by Adactio.

Next time someone asks why I’m reinventing the wheel, I’m telling them it’s because the wheel API is being deprecated. developers.google.com/chart/terms
— Jeremy Keith (@adactio) May 15, 2012

That is obviously not always possible… and is probably not the right solution either because re-inventing the wheel can be a real rabbit-hole which may take you to building a power grid!

Another solution is to favor protocols over APIs. To over-simplify things, an API is just a combination of uni-laterally decided upon protocol and an endpoint. The Twitter API is the twitter protocol used against twitter.com. Got it? This is what Status.net understood when they decided to write a compatible API… for which you would just have to change the endpoint from twitter.com to status.net. That worked until Twitter changed its API.

Now, if developers relied more on protocols, that means that when a providers disappears or deprecates a service, developers could just change the endpoint, without changing anything in their code to make sure everything keeps working. Re-using code is the most satisfying feeling for many developers.

Want an example? Last week, I got tired of myopenid.net inability to stay up. I used them as a provider for my personal openid endpoint. Changing the provider was as simple as changing the openid2.provider href in the HTML and I was done.

When you decide to work with a given’s service API, please tell them you want their API to become a protocol, something for which you are a stake holder, that you can help improve. Also, if there is any open protocol or any schema that does what this API aims at doing, please convince them that it’s way to go, that this will make the web better and that will make your code more useful to more people :)

At Superfeedr, we’re obviously big promoters of protocols (XMPP, PubSubHubbub, ATOM, ActivityStreams) over APIs, and we obviously kept the same philosophy for Msgboy.

Addendum: what makes me even sadder about this story is that Brad Fitz, who wrote the Social Graph API is the one who told me first about protocols being better than API..; and guess what? The Social Graph API has been deprecated as well. :(

Superfeedr Heroku

2012-04-12T00:00:00-07:00

For the past few months, one of our projects involved working on an Heroku addon. Today it reached public beta. In a nutshell, it allows for anyone using Heroku to integrate with all of Superfeedr’s features with minimal effort: all the provisioning, integration and even billing is managed thru Heroku.

Integrating

The first step is to add the Superfeedr addon to your existing application. Like for most Heroku addon, type this in your application’s repository:

heroku addons:add superfeedr

The next step is the most important step: picking up the right API to use:

The XMPP PubSub may require you to run an extra Dyno (process), and XMPP may not have good libraries for your language of choice.
The PubSubHubbub API uses webhooks, which means that your development machine will need to be ‘accessible’ from outside the firewall.

Then, you can start implementing! When making calls to Superfeedr, make sure you use SUPERFEEDR_LOGIN and SUPERFEEDR_PASSWORD as your credentials. (you can also set these environment variables to your local development machine)

Billing

For now, using the Superfeedr heroku addon is free, but we will soon introduce plans. These plans will obviously include a free version, but they will be based on the number of feeds (we exclude track feeds though), rather than the number of entries. This has been a rather frequent request for superfeedr and we may use Heroku as a sandbox for a new superfeedr pricing model.

If you’re looking at sample applications, check this node.js example which uses XMPP_, Superpipes, or this Sinatra applicationapp.rb, which uses PubSubHubbub.

Happy coding!

Weekend Projects: Superpipes

2012-04-08T00:00:00-07:00

Google has their 20% free time. At Superfeedr, we use weekends, or flights.
This weekend has been pretty productive, we two open sourced projects.

Superpipes

It’s a rather frequent question from new users: what’s the difference between Superfeedr and Yahoo Pipes? It’s not that easy to explain, but (at least for the feeds part), Superfeedr can be used to build something like Yahoo Pipes, and that’s what we did.

The little app we built, called Superpipes aggregates feeds in realtime, but also transforms them. It’s really simple to configure, deploy, but also hack on. It’s using node.js, express, redis, heroku, and of course, Superfeedr.

Follow the instructions if you’d like to deploy your own instance and start aggregating!

Feediscovery with node.

We sincerely love what we call community webservices. They are webservices that provide a simple feature that every developer could just implement themselves, but which are also so much more efficient if everybody shares them. Providing the favicon for any url is an example. We built feediscovery to extract urls from a url (and we’re working on that to support webfinger too!).

Now, web services are not always easy to integrate in actual apps: you have to use and HTTP library, JSON parsers… etc. Sometimes I wish I could just type google('keyord', function(err, results) { ... }); and get the results yielded, so we built node-feediscovery, and you can use it exactly like that: feediscovery('url', function(err, links) { .. });.

What’s your latest weekend project?

PubSubHubbub v0.4

2012-04-03T00:00:00-07:00

The PubSubHubbub adoption is growing strong. There is not a week where a major publisher or a major subscriber eventually implements it. Yesterday, that was NewsBlur’s turn.

At the same time, the current PubSubHubbub spec (0.3) is a bit rusty and presents a few issues that prevented more platforms to use it. The 2 most significant issues are the following: arbitrary content support, opening the door for private feeds.

We recently worked on the version 0.4 of the spec, which, solves these problems.

Arbitrary Content Support

The current spec is coupled with the Atom and RSS specs very deeply. However, in theory, PubSubHubbub shouldn’t really care about what is being transported for as long as it’s available via HTTP.

Discovery

The current spec states that discovery is performed by adding a <link> element in the “head” part of the feed (the one which contains all the meta data about the feed). Luckily, HTTP also has a “head” part: the HTTP headers.
We proposed that doing discovery for arbitrary content resources is done thru the use of Link Headers.

They look like this:

HTTP/1.1 200 OK
Date: Tue, 03 Apr 2012 08:02:19 GMT
Content-Type: text/html
Content-Length: 11261
Link: <http://pubsubhubbub.superfeedr.com>; rel="hub"  
Link: <http://blog.superfeedr.com/my-resource>; rel="self"

Please note, that similar to the 0.3 spec, we need both a self link and a hub link.
A consequence of using HTTP headers is that, instead of doing a full GET request, subscribers may just use the verb HEAD to perform the discovery of a resource.

Diffing

Feeds have a built-in diff mechanism. As they are collections of items, it is quite easy to identify what items have been added to the collection between to GET requests.

Now, using arbitrary formats means that it is not always easy or obvious to diff the resource in a meaningful way.

The 0.4 spec proposes to leave diffing up to the publisher an subscriber to identify what is the right behavior, between propagating the full document (when it was updated), or just propagating the diff, using the mechanism of their choice (text based diff, or content based diff).

Notification

When using feeds, the whole content is included in the feeds, including, as we’ve seen the discovery elements. Since most HTTP document do not have this feature, it means that the notification must include some of the HTTP headers served with the content, including, well, the Link headers.
This is an important difference with the 0.3 spec, and most not be overlooked, as some resources have meaningful information in the HTTP headers.

Signature

The only way to perform a secure notification and avoid “man-in-the-middle” attacks, it is important that the notification from the hub to the subscriber is signed, using an HMAC signature with a shared secret. This behavior is the behavior used in the version 0.3 of the spec.

In the version 0.4, we use the same mechanism, but please note that this will only sign the content, and not the headers. As such, if a subscriber wants to trust the complete notification, the only option is to use an HTTP*S* callback url.

There is another way, which is to provide a specific header which lists the signed headers, but we believe it’s overly complex. Please let us know if you disagree.

Private Resources

This is the other key aspect of this revised spec. Currently, the publisher doesn’t have any visibility on ‘who’ the subscribers are (and if there are any actually).

This is a very strong problem in the context of social networks where someone may want to distribute its content only to a restricted set of people: their friends, or family.

Typically, they want to approve or deny subscriptions, but also may want to later reject a previous approved subscription. They may also want to limit the subscription to a subset of the actual published feed.

Subscription

The current spec only uses the verification of intent to determine if a subscription should be accepted. We now need to use the publisher’s input to accept or deny subscriptions. It is also very likely that the publisher will wait for the actual user’s input before accepting or denying, which means that the subscription process now becomes asynchronous by default.

Publisher Verification

This is a brand new step, but it remains optional and subscribers should completely ignore this part. Using an agreed upon mechanism, the hub will ask the publisher whether the subscription should be accepted, by echo-ing the subscription request. The publisher can then decide to accept or deny the subscription, as well as include a reason for that.

This mechanism can, for example be combined with the use of a From HTTP header in the subscription. Then, using a webfinger discovery mechanism, the publisher can identify the actual user behind the subscription. Other mechanisms could also very well be used, but are beyond the scope of this spec.

Subscription Validation

Once the subscriber’s intent has been confirmed AND if the publisher agrees to the verification, then, the subscription should be validated and the hub will send a request to the subscriber to validate that subscription.

If not, the hub should also send a request to indicate denial, with, if applicable, the reason used by the publisher. The subscriber may then either retry (using additional parameters) or just abandon its subscription.

The subscription validation process can also happen later to deny an existing subscription.

Conclusions

The draft 0.4 clearly still has a points that needs to be clarified and improved, but we believe it solves the 2 current major issues that the current PubSubHubbub protocol has.

It is obvious that if a hub doesn’t respond to the 0.4 spec API calls, a subscriber should fallback to 0.3 for as long as the 0.4 spec hasn’t been widely adopted.

Please, send your feedback.

Flattr PubSubHubbub-ed

2012-03-09T00:00:00-08:00

I have to confess I am not the best at writing consistently and often on that blog. There are so many topics that I want to write about: writing a Chrome application, the future of PubSubHubbub, not using JQuery when it’s not necessary, our Heroku Addon… etc. But today, I want to share with you an awesome new hub that we host at Superfeedr.

Flattr

Flattr is a very innovative service that brings a very cool monetization solution to anyone creating web content. Each user decides on how much money they want to give to the internetz. During the month, they can then flattr various things, including music, video, games, software, or even people! When the end of the month comes, the user’s money is shared between all the thing he previously flattred.

PubSubHubbub

Flattr provides a lot of different feeds, and they’re now all PubSubHubbub! It becomes really easy to resyndicate the things you’ve flattred using a user activity feed. A thing owner could also subscribe to the feed of people flattring it… and then send a realtime thank you tweet :p

We’re extremely proud to see services like flattr use PubSubHubbub, as we know their community is clearly among the trend setters. Who’s next?

Grove.io and Superfeedr

2012-02-19T00:00:00-08:00

Do you know Grove.io? It is a great hosted IRC service with a bunch of really awesome features to make your rooms much much smarter and useful. One of the great things they did is allow their users to plug other applications to the room using webhooks.

The obvious use case for a team of developers is to show commit messages, but we wanted to be able to post the content of an RSS feed to a room very easily as well. A team can just use that to plug the comment feed of the team’s blog, or the feed from their error collecting application (airbreak.io has RSS feeds for each project)… etc.

Enter Supergrover. It is a simple node.js application (less than 200 lines of code!), which allows you to enter the url of any RSS feed, as well as the url of your grove.io room’s callback, so that when the feed updates, a message is posted to your room.
It doesn’t even need to store data, as it uses callback urls to store the state of the subscription (feed and channel).

It’s a great use case for our Heroku Add-on, as well as the proof that one can build applications quickly using PubSubHubbub and Superfeedr using node.js :)

Sinatra, Heroku and Superfeedr

2012-01-21T00:00:00-08:00

This is a short tutorial on how to deploy an Sinatra web app that uses Superfeedr to Heroku. This app provides a very simple home page that lists the latests entries of some of your favorites sites. It’s greatly inspired by the awesome start.io from the awesome Peter Vidani and Jacob Bijani

Set up

We will use Sinatra, the Rack Superfeedr gem, as well as Twitter’s bootstrap for the layout, because I suck at making things shinny.

Let’s start first by creating the application on Heroku.

Create the repo:
mkdir start-page && cd start-page
git init
git add .
git commit -m "init"

Create the application on Heroku:
heroku create --stack cedar

Add the hostname as an environment variable. It is used by the rack middleware to build the callback urls:
heroku config:add HOST=<YOUR APP DOMAIN>
Add the superfeedr addon:
heroku addons:add superfeedr

Implementation

This implementation is minimalistic, feel free to look at the code closely.

Let’s add the files:
touch app.rb
touch config.ru
touch Gemfile
mkdir views/ && touch views/index.erb

In app.rb:

In config.ru:

In Gemfile:

In views/index.erb:

git add .
git commit -m "implemented" -a

Deploying

That’s the simplest part:

First, let’s install the gems.
bundle install
git add Gemfile.lock
git commit -m "adding Gemfile.lock" -a

And push the code
git push heroku master

… And that’s it!

Point your browser to your Heroku application. It is likely that you’ll have to wait for the feeds you have added to have new content before you see it appear on your home page. The last step is to make this page your default start page in your favorite web browser.

Check mine!

Gotchas

Heroku will put this application to sleep if you only use a single dyno… and since we store the entries in memory for now, that means you’ll lose that data if the app goes idle. The solution is very simple: store the data, using another Heroku addon!

Of course, this is a very simple application and there is a ton of little things that one can do to improve it:

add the ability to dynamically subscribe to feeds
show meta data, like the number of bit.ly clicks on each of the stories
auto-refresh the page
…

The Superfeedr Heroku addon is still in alpha to this date, so if you want to run, this, please email us and we’ll get you in!

Rack Middleware for Superfeedr

2012-01-19T00:00:00-08:00

I am a bit ashamed that this comes so late… but it’s always better late than never: we have a rack middleware for Superfeedr.

What it does is simple: subscribe, unsubscribe and help you handle notifications by hiding all the PubSubHubbub ~~complexity~~ magic. Since it’s Rack, it should work with any Ruby web framework that supports Rack, including Rails and Sinatra.

Get it while it’s hot: gem install rack-superfeedr, and check the code on github. If you build something awesome, please let us know and we’ll link to it.

As you can see it’s quite elegant :) Can’t wait to show you how to run that with our Heroku Addon! if you want to be among the very first testers. Please let us know!

Publish Subscribe for the web

2012-01-13T00:00:00-08:00

Everybody tries to make the web better. Here’s my attempt.

A brief history of the web.

READ

I’m 29, but I was lucky enough to be among the very first web addicts. When I remember most of the websites from that time, I am pretty embarrassed by how ugly they were, but also (worse) by how useless they were. I’m sorry to say, but at the time, the web was not making anyone’s life better. The only thing it did was replacing shinny paper based magazines with ugly looking screens. I now fail to see how I could consider this as an improvement.

A lot of people wrote this, but the web was mostly READ ONLY. In other words, aside from a very few things, the only activity one could ever do was read. This is why we have pages, bookmarks, site indices… etc. At this time, search engines appeared, because people needed to a way to find the content they wanted to read. Similarly e-commerce was born, because quickly, many companies exposed the traditional product lists on the web so that their prospects could read them.
This behavior is baked deep into the protocol that we use to access the web, in the form of GET requests.

WRITE

A bit more than 10 years ago, a massive shift happened: we started to write to the web. That was interesting. For once, the web was offering something new as very few people were actually able to write in newspapers and magazines! Blogging was the very first step of this and quickly the web sites that allowed us to do write saw the massive spike in traffic: wikipedia, youtube, flickr outgrew the old portals that ISPs were forcing down their user’s throats.
It’s also at that time that API started to appear. Developers were able to program services that would write to 3rd party websites, instead of actual people. Writing is also baked deep into our protocols in the form of POST requests.

People were now able to read and write the web. Increasingly they were interested by what other people they knew wrote, because, after all, that is who they used to interact with in the real world. It slowly put people in broad light, as opposed to big media outlets. The logical next step was to map our physical friendships to the web. That’s the social networks wave. The real new activity here is this bounding: expressing interest in someone to later read the content they wrote. Whether you call that subscribing, following, friending or circling, it’s all the same pattern.

Interestingly enough this is more than just social. Several very successful ecommerce services offer the same kind of pattern and allow you to subscribe to their deals mailing lists. Coincidentally, we now access the web more through mobile devices than through our computer, which means that context (place and time) is increasingly important, and some services are able to provide us a better experience by subscribing us to our context. What’s interesting is that despite the obviousness and the ubiquity of this subscribe pattern, there is no such verb into the HTTP protocol.

Hooks and Callbacks

The little story above is nice, but, even if we don’t have this “subscribe” verb, it looks like we do already have all the services working just fine, so why would we need something to do it?

The first answer is obviously standards. By having a standard way to doing so, we could probably decouple that action from specific services. If that happens, it means that we could start using generic clients to subscribe to all kinds of things online. Do you use a different browser for Facebook and Groupon? By pushing down the stack the subscription pattern, we can abstract it! Hopefully we can even bake this in browsers in one way or another.

Another reason is that I hope and believe that this will open the door for a new way of creating web applications. Several programming languages also have a Publish Subscribe pattern. These languages and frameworks are usually very efficient and elegant for decoupled systems. Many modern web technologies like Node.js, but also datastores like Redis’s PubSub have made it much easier to write complex workflows.
Plugging complete web applications will certainly make creating complex systems much easier. Most web developers don’t think much about electricity, cooling or even IO, with the advent of fully managed platforms. Layering common functionnalities and building on top is an amazing enabler.

For example, not only could we (humans and machines) “subscribe”, and combine that with other actions, like write. We could even imagine machines subscribing to other machines’s action to let the user decide which workflow he wants. Does this remind you of WebIntents? it should, as web intents are a way to subscribe to a user’s actions. I told you this publish/subscribe pattern is everywhere!

Getting involved!

Some of the smartest web engineers have fought for years to bring this subscribe approach to the web stack, through start-ups, the use of other non-http protocols, by bending HTTP, or by writing HTTP-based protocols for specific sub-cases. None of these approaches became as ubiquitous as the pattern itself.

But today, there is some kind of momentum as the web’s data is becoming harder and harder to ingest for everybody (machines and humans). This is why I’m trying to push the W3C to sit down and take some time to think about all this. I have created a Community Group there and I hope that slowly, but surely people will express their needs, and why having this would make the web better. There are number of topics and use cases that need to be covered, I hope and believe we will cover them all.

Please, join us.

Twitter SERP Drama

2012-01-10T00:00:00-08:00

Today Google introduced significant changes to their algorithm. Like every time this happens some people are scared their free pension will end. This is clearly due to Google’s ~~too~~ incredibly high market share. This time, among the whiners, is Twitter, and this is interesting.

Not so long ago, Twitter sold its public and user-owned [their words, at the time] data to Microsoft and Google. What’s interesting is that this is, to my knowledge the only time that Google paid anyone to index their content. A couple years later, it looks like Google has sobered down and they eventually did not renew the Twitter contract when it ended. In other words, Google considered that Twitter’s data wasn’t worth more than anyone else’s data on the web and that their regular crawling should be plenty. Now, Twitter seems to be a bit sour, because not only have they lost a significant revenue opportunity, but they also likely lost couple positions in Google’s search engine result pages.

The $1Bn question becomes : did Google favor their own results over Twitter’s or did Twitter just go back to where they belong? Some people may give Google the benefit of the doubt after last week’s Chrome-gate, but that’s not enough. The only way to know for real would be to make sure that Google+ and Twitter have the same visibility and that Google can access Twitter’s data in the same way that they access their own data.

This is where open protocols win. Google has been a strong advocate for an open web, with no barrier, no contract and an undiscriminated access. If Twitter supported open protocols, Google (and anyone else) would be able to get this public data in realtime, and they should then get their old SERP ranking back. If they did not, well, then, we could certainly suspect that Google is pushing up its own results.

Using proprietary protocols, forcing partnerships, discriminating between services mostly obfuscates the play field. At the time nobody was really shocked that twitter and Google entered in a specific agreement, but what we’re seeing today is the immediate consequence. Maybe Google is favoring their own results, but even if they did, can Twitter really complain about it? They lost their credibility when they restricted the access to that data: Nemo auditur propriam turpitudinem allegans