Superfeedr Blog : Real-time cloudy thoughts from a super-hero

Sinatra, Heroku and Superfeedr

2012-01-21T00:00:00-08:00

This is a short tutorial on how to deploy an Sinatra web app that uses Superfeedr to Heroku. This app provides a very simple home page that lists the latests entries of some of your favorites sites. It’s greatly inspired by the awesome start.io from the awesome Peter Vidani and Jacob Bijani

Set up

We will use Sinatra, the Rack Superfeedr gem, as well as Twitter’s bootstrap for the layout, because I suck at making things shinny.

Let’s start first by creating the application on Heroku.

Create the repo:
mkdir start-page && cd start-page
git init
git add .
git commit -m "init"

Create the application on Heroku:
heroku create --stack cedar

Add the hostname as an environment variable. It is used by the rack middleware to build the callback urls:
heroku config:add HOST=<YOUR APP DOMAIN>
Add the superfeedr addon:
heroku addons:add superfeedr

Implementation

This implementation is minimalistic, feel free to look at the code closely.

Let’s add the files:
touch app.rb
touch config.ru
touch Gemfile
mkdir views/ && touch views/index.erb

In app.rb:

In config.ru:

In Gemfile:

In views/index.erb:

git add .
git commit -m "implemented" -a

Deploying

That’s the simplest part:

First, let’s install the gems.
bundle install
git add Gemfile.lock
git commit -m "adding Gemfile.lock" -a

And push the code
git push heroku master

… And that’s it!

Point your browser to your Heroku application. It is likely that you’ll have to wait for the feeds you have added to have new content before you see it appear on your home page. The last step is to make this page your default start page in your favorite web browser.

Check mine!

Gotchas

Heroku will put this application to sleep if you only use a single dyno… and since we store the entries in memory for now, that means you’ll lose that data if the app goes idle. The solution is very simple: store the data, using another Heroku addon!

Of course, this is a very simple application and there is a ton of little things that one can do to improve it:

add the ability to dynamically subscribe to feeds
show meta data, like the number of bit.ly clicks on each of the stories
auto-refresh the page
…

The Superfeedr Heroku addon is still in alpha to this date, so if you want to run, this, please email us and we’ll get you in!

Rack Middleware for Superfeedr

2012-01-19T00:00:00-08:00

I am a bit ashamed that this comes so late… but it’s always better late than never: we have a rack middleware for Superfeedr.

What it does is simple: subscribe, unsubscribe and help you handle notifications by hiding all the PubSubHubbub ~~complexity~~ magic. Since it’s Rack, it should work with any Ruby web framework that supports Rack, including Rails and Sinatra.

Get it while it’s hot: gem install rack-superfeedr, and check the code on github. If you build something awesome, please let us know and we’ll link to it.

As you can see it’s quite elegant :) Can’t wait to show you how to run that with our Heroku Addon! if you want to be among the very first testers. Please let us know!

Publish Subscribe for the web

2012-01-13T00:00:00-08:00

Everybody tries to make the web better. Here’s my attempt.

A brief history of the web.

READ

I’m 29, but I was lucky enough to be among the very first web addicts. When I remember most of the websites from that time, I am pretty embarrassed by how ugly they were, but also (worse) by how useless they were. I’m sorry to say, but at the time, the web was not making anyone’s life better. The only thing it did was replacing shinny paper based magazines with ugly looking screens. I now fail to see how I could consider this as an improvement.

A lot of people wrote this, but the web was mostly READ ONLY. In other words, aside from a very few things, the only activity one could ever do was read. This is why we have pages, bookmarks, site indices… etc. At this time, search engines appeared, because people needed to a way to find the content they wanted to read. Similarly e-commerce was born, because quickly, many companies exposed the traditional product lists on the web so that their prospects could read them.
This behavior is baked deep into the protocol that we use to access the web, in the form of GET requests.

WRITE

A bit more than 10 years ago, a massive shift happened: we started to write to the web. That was interesting. For once, the web was offering something new as very few people were actually able to write in newspapers and magazines! Blogging was the very first step of this and quickly the web sites that allowed us to do write saw the massive spike in traffic: wikipedia, youtube, flickr outgrew the old portals that ISPs were forcing down their user’s throats.
It’s also at that time that API started to appear. Developers were able to program services that would write to 3rd party websites, instead of actual people. Writing is also baked deep into our protocols in the form of POST requests.

People were now able to read and write the web. Increasingly they were interested by what other people they knew wrote, because, after all, that is who they used to interact with in the real world. It slowly put people in broad light, as opposed to big media outlets. The logical next step was to map our physical friendships to the web. That’s the social networks wave. The real new activity here is this bounding: expressing interest in someone to later read the content they wrote. Whether you call that subscribing, following, friending or circling, it’s all the same pattern.

Interestingly enough this is more than just social. Several very successful ecommerce services offer the same kind of pattern and allow you to subscribe to their deals mailing lists. Coincidentally, we now access the web more through mobile devices than through our computer, which means that context (place and time) is increasingly important, and some services are able to provide us a better experience by subscribing us to our context. What’s interesting is that despite the obviousness and the ubiquity of this subscribe pattern, there is no such verb into the HTTP protocol.

Hooks and Callbacks

The little story above is nice, but, even if we don’t have this “subscribe” verb, it looks like we do already have all the services working just fine, so why would we need something to do it?

The first answer is obviously standards. By having a standard way to doing so, we could probably decouple that action from specific services. If that happens, it means that we could start using generic clients to subscribe to all kinds of things online. Do you use a different browser for Facebook and Groupon? By pushing down the stack the subscription pattern, we can abstract it! Hopefully we can even bake this in browsers in one way or another.

Another reason is that I hope and believe that this will open the door for a new way of creating web applications. Several programming languages also have a Publish Subscribe pattern. These languages and frameworks are usually very efficient and elegant for decoupled systems. Many modern web technologies like Node.js, but also datastores like Redis’s PubSub have made it much easier to write complex workflows.
Plugging complete web applications will certainly make creating complex systems much easier. Most web developers don’t think much about electricity, cooling or even IO, with the advent of fully managed platforms. Layering common functionnalities and building on top is an amazing enabler.

For example, not only could we (humans and machines) “subscribe”, and combine that with other actions, like write. We could even imagine machines subscribing to other machines’s action to let the user decide which workflow he wants. Does this remind you of WebIntents? it should, as web intents are a way to subscribe to a user’s actions. I told you this publish/subscribe pattern is everywhere!

Getting involved!

Some of the smartest web engineers have fought for years to bring this subscribe approach to the web stack, through start-ups, the use of other non-http protocols, by bending HTTP, or by writing HTTP-based protocols for specific sub-cases. None of these approaches became as ubiquitous as the pattern itself.

But today, there is some kind of momentum as the web’s data is becoming harder and harder to ingest for everybody (machines and humans). This is why I’m trying to push the W3C to sit down and take some time to think about all this. I have created a Community Group there and I hope that slowly, but surely people will express their needs, and why having this would make the web better. There are number of topics and use cases that need to be covered, I hope and believe we will cover them all.

Please, join us.

Twitter SERP Drama

2012-01-10T00:00:00-08:00

Today Google introduced significant changes to their algorithm. Like every time this happens some people are scared their free pension will end. This is clearly due to Google’s ~~too~~ incredibly high market share. This time, among the whiners, is Twitter, and this is interesting.

Not so long ago, Twitter sold its public and user-owned [their words, at the time] data to Microsoft and Google. What’s interesting is that this is, to my knowledge the only time that Google paid anyone to index their content. A couple years later, it looks like Google has sobered down and they eventually did not renew the Twitter contract when it ended. In other words, Google considered that Twitter’s data wasn’t worth more than anyone else’s data on the web and that their regular crawling should be plenty. Now, Twitter seems to be a bit sour, because not only have they lost a significant revenue opportunity, but they also likely lost couple positions in Google’s search engine result pages.

The $1Bn question becomes : did Google favor their own results over Twitter’s or did Twitter just go back to where they belong? Some people may give Google the benefit of the doubt after last week’s Chrome-gate, but that’s not enough. The only way to know for real would be to make sure that Google+ and Twitter have the same visibility and that Google can access Twitter’s data in the same way that they access their own data.

This is where open protocols win. Google has been a strong advocate for an open web, with no barrier, no contract and an undiscriminated access. If Twitter supported open protocols, Google (and anyone else) would be able to get this public data in realtime, and they should then get their old SERP ranking back. If they did not, well, then, we could certainly suspect that Google is pushing up its own results.

Using proprietary protocols, forcing partnerships, discriminating between services mostly obfuscates the play field. At the time nobody was really shocked that twitter and Google entered in a specific agreement, but what we’re seeing today is the immediate consequence. Maybe Google is favoring their own results, but even if they did, can Twitter really complain about it? They lost their credibility when they restricted the access to that data: Nemo auditur propriam turpitudinem allegans

Node Superfeedr

2012-01-06T00:00:00-08:00

It’s long overdue, but we built a little node.js library to interact with Superfeedr from any node.js application. I want to thank Martin Borho for his precious help, as well as Astro for his amazing work on node-xmpp

A bit of theory

We built this wrapper using our XMPP API. This may sound surprising to a lot of you, as I have advised to a lot of people to use PubSubHubbub when they were building a web app.

The main reason behind this is that Node.js is the perfect framework to integrate an XMPP client, transparently. For most people the syntax behind any node application is a bit odd with all these callbacks. It simply comes from the fact that Node.js is an implementation of the very famous reactor pattern. This means that node.js can handle stateful protocols (like XMPP) very well… Most of web framework do not offer this, because they map HTTP’s stateless’s approach.

Installing

NPM is now a first class citizen in node… which means you can install superfeedr-node with a simple:

npm install -g superfeedr

Usage

In your node application, just instantiate a new Superfeedr Client, connect it, subscribe to feeds you need, and handle incoming notifications. It should all be relatively self-explanatory.

FAQs!

Will this work behind the firewall?

Yes! As opposed to PubSubHubbub, the connection initiated by the client is kept alive (XMPP FTW!), so Superfeedr can just use this connection to send you the data :)

Will this work on fully-managed platforms (Nodejitsu, Heroku…)?

Yes! Of course… this should work very well!

It sucks, can I make it better?

Please do, fork it!

Happy New Year

2012-01-05T00:00:00-08:00

2012 is starting and, well, I have to confess we haven’t published as much as we wanted on this blog! This is one of our new 2012 resolutions!

Have a wonderful year, full of amazing projects, using technologies that you love!

We have a LOT on our plate… and I can’t wait to show you everything we’ve been working on ~~secretly~~ over the past couple weeks :)

WebIntents

2011-10-25T00:00:00-07:00

We hate barriers and user locks of any kind, but the web is full of them. The web works better when services are decoupled, when the user is at the center and picks the services he wants to use, not the services other services wants him to use.

One of the key characteristics of the msgboy is that it integrates seamlessly with your web : the user’s web. We have 2 features in mind : allowing our users to subscribe to more content using the msgboy or 3rd party services, and allowing the msgboy’s users to share the content they received using their favorite 3rd party services.

Enter WebIntents.

Paul Kinlan is a developer advocate at Google. He introduced WebIntents almost a year ago. WebIntent’s general idea is to allow web services to register features with each others in a decoupled way. One of the example that Paul uses often is editing an image to remove red-eye.

If I am the developer who creates a photo library online, I may not have that feature. Yet, I know there exists services online that do that very well (maybe better than what I would do). WebIntents allows these 3rd services to register their “red-eye-removal” feature as an “intent”. WebIntents also allows my app to call these services without “knowing” about them previously.

If you think about it, the web is full of services that do a great job, but the web itself would be tremendously more useful if all these services worked together in workflows.

Msgboy wants to be a first class citizen of the web by allowing you to subscribe to content thru it anywhere you are. There are 2 ways of doing it :

Convince every other website out there to just work for us and add a msgboy button to their site. Of course it’d be much better for us in terms of branding…. but that wouldn’t work. Why would people promote us?
The other option is to implement the WebIntents so that rather than promoting another company/service/product, they promote a feature and make their application more useful for their users, without trading them!

Registering the “subscribe” intent.

Registering an intent is fairly easy; you add the following code to your web application’s HTML code (in the <head> part) :

<intent action="http://webintents.org/subscribe" />

Also, make sure you add a <link> element to webintents.min.js

What this code does is simple : it registers an intent on the webintents.org service. This intent is to handle the “subscribe” action on behalf of the user. There are a couple more options, but for the sake of simplicity, let’s stick with this simple default : feel free to check the msgboy’s webintent’s branch to see how registration is performed in the context of a more complex app.

Technically, what happend here is that webintents.org stored the intent (using HTML5’s localStorage). Which means that all the user data is local, but we’ll come back to this a bit later. Of course the action/verb is stored, but also the url of the page that declared the intent.

Starting the activity.

Say you have a blog and you want to provide a subscribe button that your readers could use to indicate they want more of your content, using the application of their choice. When the button is triggered, you would do something like this:

var intent = new Intent("http://webintents.org/subscribe", "application/atom+xml", "http://blog.msgboy.com/rss"); window.navigator.startActivity(intent);

What this does is quite simple: it creates an action of type “subscribe”, and uses the params "application/atom+xml", "http://blog.msgboy.com/rss" for it. It then starts the activity, and that’s when the magic happens:

If you looked a bit in the webintents.js code, you would see that this will just send a request to webintents.org to perform that action.
Webintents.org will then open a popup and read the previously stored data in local storage about intents.
If there intents registered for that action, they are presented to the user
If the user choses any of the previously registered application, it it redirected to it.

If you implement WebIntents you have to know that the params for the action are stored in a base64 encoded JSON stringed used as the window’s name.

What’s next

It is fairly easy to register events, but also to trigger actions, and that works fine, even though this is very early for webintents. However, ff you’ve taken a couple minutes to think about it, you will rightly notice that webintents.org becomes a single point of failure, which kind of kills the principle of decentralized, decoupled, that’s so precious for the web.

Of course, you can run your own own if you forked the code from Github. But then, if an app is registered on a given web intents server, while the action is triggered on another server, the chain will not be completed.

Luckily there is a solution for this, and that was actually Paul Kinlan’s orginal idea : use the browser to register intents and trigger actions. Browsers already include a lot of Javascript APIs, and it’s Paul’s goal to have them integrate the webintent’s API as well. When that happens there is no need to use a 3rd party server. Users will eventually use their browsers to register intents and trigger actions, and the workflow stay the same.

Krtconf (realtime web conference)

2011-10-18T00:00:00-07:00

We’ve been a bit silent over the past couple weeks, but we have an excuse: days last 24 hours! We worked hard on the msgboy. Chrome’s latest versions broke our legs several times (changes in IndexedDB, changes in Websockets… etc). If what doesn’t kill us makes us stronger, Msgboy is going to be super-hero strong.

Hopefully we can introduce it very soon, at the Krtconf.

Krtconf is a 2 day conference (Nov 7+8), organized by developers, for developers, in Portland, OR. It’s the place to be to stay to date on all realtime web technologies : XMPP, PubSubHubbub, Websockets, Socket.io, long-polling, and many many more. You will learn about these, but you will also get to implement them, with lots of demos, hackathons and conversations.

Check the list of speakers. A lot of them are heroes for us. You’ll see that you can’t miss such an event, so grab your ticket now ($300 discount if you book before Oct 21st).

Superfeedr is co-organizing it with the great folks at &yet. We have a few speaker spots left, for sponsors mostly. If you’re interested, please drop us a note.

Superfeedr on FLOSS weekly

2011-10-13T00:00:00-07:00

Twit is an amazing web success. It’s been an honor to be invited to talk about Superfeedr and PubSubHubbub on FLOSS Weekly!
Watch the show :

And don’t forget to subscribe to the podcast, as it’s one of the best podcasts out there :)

Node XMPP Server

2011-08-26T00:00:00-07:00

At Superfeedr, we’ve always been great fans of node.js. We believe the ability to use the same language on both servers and clients will breed a whole new generation of web developers that can do amazing work on both ends. Last year already, we released node.js code a couple hours ahead of NodeKO, and this year, we’re doing it again.

A node KO treat

In the context of the msgboy (request your invite now!), we had to setup a new XMPP server. After fighting with ejabberd, we installed Prosody, but we also quickly concluded that we would need a lot of customization, as well as the ability to tweak our server(s) to fit our exact needs.

Since astro had been doing an amazing work with his node-xmpp library to build Client, Components and even Server to Server modules, the logical next step was to try to build a Client to Server module so that we could have a full blown server. That’s what we worked on the past couple days, and it’s now on Github!

Beware though… as it’s extremely early and some key features are not yet supported. Our biggest challenge is trying to fix a weird TLS bug. Please help if you can!

Let’s get dirty

If you’re brave enough and want to get started, install our branch of node-xmpp from source. You should then be able to run a server by typing
$ node examples/c2s.js

Now, take your favorite XMPP client (we used Psi, but you need to make sure it will accept to transmit PLAIN text passwords over an unencrypted connection) and connect to the server as user@zipline.local. Use password for the password.

Now, that was ~~relatively~~ easy, and quite useless so far, as you could do that with any XMPP server out there. The real interest of this one is that we wanted to give full control back to the developer to customize their server as much as they needed. Take a look at the examples/c2s.js file for the code that runs this specific server:

Auth is simple: the c2s module will just callback with a username and a password. Your code should then decide whether you want to authorize or not connection, based on your database, or any other piece of application that you want.

You can define modules supported by your server from the most popular XEPs, using node-xmpp’s mixins. The modules should be at the heart of any implementations, as we want to keep the c2s code itself very bare.

There are currently 4 events that you can listen to on the C2S module itself : connect, disconnect, authenticate, stanza, which are relatively self explanatory. The connect event should be used to register mixins to user connections. The stanza event should probably be left empty (unlike in the example), as all meaningful stanzas should probably be treated in mixins.

Clustering!

Also, you get all the JS magic that lets you rewrite any code without much hassle. One of the key aspects of most XMPP servers is their -in-ability to be deployed in clusters to handle more users. For those who want to dive in that direction, Redis’s PubSub provides an elegant solution. Combining this with some sharding on the jids should probably go a long way! You can see in the basic example below that we rewrote the routing mechanism to use Redis and run multiple instances of the same server in a cluster.