Superfeedr Blog : Real-time cloudy thoughts from a super-hero

This week in Google - Episode 52

2010-07-26T00:00:00-07:00

If you’re not listening to any of Leo shows, you’re certainly missing a few of the best online shows of these days. Leo and his hosts get the web and the last episode to date of ThisWeekInGoogle (happy birthday) is just the best proof.

Jyri was on last week’s show and he did a great job explaining Superfeedr, but this is really one of the many awesome topics in this podcast. Don’t miss the part on data privacy, the part on data resyndication. I think they are spot on.

Watch it in video, or subscribe in iTunes.

Internet of Things and PubSubHubbub

2010-07-22T00:00:00-07:00

The Internet of things is one that makes us dream : everything connected to the internet, talking together and making our lives easier. It’s something that I’ve been thinking to a lot lately.
This all comes from this weird dream that I made, that I would be friended on Facebook by things I own ; my iPhone, my pair of sunglasses… etc. After all, they’re probably interested in me as much as my 400+ friends (Love you guys!).

If my bed subscribed to me on gowalla, it would warm up before I even reach it. If my bathroom mirror was my facebook friend, it could certainly tag me every morning and make a pretty cool time lapse. We have all these awesome social services, but none of them is made for objects. This would yet be quite easy : each object would have a profile, and would have an activity stream, based on what it does and to which I could also subscribe. My iPhone’s stream would be something like this :

Mom called
Call dropped ~~oups~~
Moved to Coffee Bar
Connected to Wifi
Battery down to 30% ~~oups~~
Mating with MacBook Pro, refilling
Unplugged
Playing Spotify
Sound up

You get the thing, right?

Not that it’s all interesting, but still, once there is this stream, I can certainly subscribe to it… and learn a bit more about my new object friend, like for example, get a notification on my laptop when its battery dies while in my coat’s pocket, or even find it if it’s been stolen (wait, you can already pay for that). I can also learn a bit about myself : how much music do I listen to, and even send an email automatically to sjobs@apple.com for each dropped call! The iPhone looks easy, but imagine the same thing about my pants :

Being washed
Being ironed (no idea who would do that)
Worn, for 12 hours
Worn, for 10 hours
Temperature dropping
Worn, for 12 more hours
Worn 4 times: ~~gross!~~

Now, my pants would be smart enough to know when I need to actually wash them? Even better, let’s imagine that my heating system can subscribe to my pants, so that when my pants are cold, the heating system goes on? How cool would that be (except for those of us who don’t wear pants.).
What about my car subscribing to my home on foursquare? So that when the car checkins the house, the garage door opens.

We’ve been talking about home automation for decades. No idea why this didn’t really take off, but I’d bet that the fact that all these use proprietary protocols and data structure doesn’t help. Luckily, we now have a full protocol stack for all this : Atom, ActivityStreams, GeoRSS to represent the data, HTTP for the communication, and PubSubHubbub for the subscription mechanism. Time for a shift!

Google Buzz Firehose

2010-07-19T00:00:00-07:00

Our track feature brought us a lot of new subscribers. Track is definitely one of our big priorities of the moment as people love it! One of the primary concerns, is to increase its search base. 20M+ entries per day might not be enough for some of you, who want even more data. Starting today, you can use Superfeedr’s track feature to get realtime PubSubHubbub notifications from Google Buzz for any entry that matches a given keyword.

Obviously, we’re adding more and more PubSubHubbub content to Superfeedr so you can build awesome Track-based applications!

Also, don’t forget, you can get this feed and any track feed either thru XMPP or PubSubHubbub =D

A Better track

2010-07-16T00:00:00-07:00

It seems that you people love our track feature. Much more than we anticipated, so we’ll be focusing on this in the coming weeks to make it both smarter and better. The version 1.0 is great, but many of you asked for the ability to track more than just 1 keyword, some of you have asked to track data from a specific source, or within certain geo boundaries. We want to work on this and I’m preparing a few requirements and design options.

However, it would be very interesting for us if you could make your feature requests for this track here, so we can prioritize and see what you guys truly need.

Operands

You want combinations of keywords. Which ones?

AND or + : the entry will have to match both keywords independently

OR or | : the entry will have to match either keywords. We will probably not support this, as it’s really the same as subscribing to 2 feeds.

"" : the entry will have to match the exact sequence between the quotes

- : the entry will have to match 1 word (before the minus), but not the other (after the minus).

Any other thing come to mind?

Filters

Sometimes, instead of all the data coming from Superfeedr, you may want to filter it.

By feed : You want all the entries in a given feed that matches a given query.

By publisher/hub : You want all the entries from a given publisher that matches a given query.

A location : all the content within a radius of a given location. This could be combined with a query.

DOM based : our schema is XML. Some people may want to subscribe to all entries that have a category “Apple” and include a link that has a rel=“reply”.

I don’t have any ETA for this, but that will certainly be our focus from now on. Please tell us if you have specific needs.

More data

We’re working on extending the amount of data that flows thru Superfeedr. Our subscribers add more content every day, and we’re trying to work with big publishers to let you filter thru this data as well. If you’re a publisher and want to have your data accessible thru our track feature, like Posterous, Tumblr, SixApart, Gowalla, HuffPost, GawkerMedia and more than 900 other publisher, please, get in touch!

The web dissolving

2010-07-06T00:00:00-07:00

In the past few weeks, I’ve been struck by the thought that the web is dissolving. There used to be a time (like 6 years ago!) where we could only browse the web in specific places, from specific devices, at specific times and then, we would go on specific sites.

The web is dissolving in geographical space

In the early days of the internet, we would talk about the connected people at home or at work. When we used to go to my parents little ski studio in the french Alps, there was no internet, not even in the “village”. Today, they’re thinking about wiring up the whole building on the same basis as they put water or electricity in it.
Most of the commercial flights over the US now offer Wifi! I’m not even talking about all these new shiny smart phones, but basically, the number of places where you cannot be connected is shrinking like crazy.

The web is dissolving into time

When I was a kid, we would have a limiter number of “computer” and “tv” per week. The goal was obviously to open up to the rest of the world. I’m not sure about today’s kids, but what I know is that I kind send or receive emails at any time, fact check at any time of the night… There is no time for some web browsing anymore.

The web is dissolving from our computers

5 years I ago, I had at most 2 applications on my computer which would be “web ready”, which would be able to communicate with other apps outside of my computer : my web browser and my email client.
At this moment, out of the 12 apps that are running on my laptop, 9 of them do actually send and get data from the web (at least explictly!). At the same time, my OS itself keeps getting update from “the cloud” and many of my desktop apps sync data with existing web services : calendars, contacts, pictures or even files, with apps like Dropbox. It even goes further than this, with more and more devices connected all the time to the internet (and sometime, devices who couldn’t work without some kind of connection to it, like fitbit). There is no more single internet endpoint.

The web is dissolving from our websites

This I think is the most intriguing fact. There used to be a time where I would be on NYT.com, on Craigslist, or even on my wells fargo account. This is dissolving too : the data I consume is more and more decoupled from the publishing site. I very rarely go to NYT.com’s home page, however, I keep consuming it from links I see passing by, from news reader… etc. My bank account? I access it from Mint.

The next step is to actually have a true data freedom, where not only my data is exportable, but also where it is in sync : I update my status on service Y, and service X knows about it. I add a friend on Facebook and Flickr knows about it too. My web profile will become this “bus” of data about me that people can access from several endpoints (like Twitter, or LinkedIn, or this blog), and not a huge archipelago of small bits that I need to update one by one.

Track

2010-07-01T00:00:00-07:00

Ever wondered how you could subscribe to keywords, instead of actual feeds? It’s already great to get pushed content from thousands of RSS feeds, but sometimes, you want to know all about your company’s name, all about movie that’s just been released. A long long long time ago, Twitter had its own track. It was basically Twitter search inside your twitter stream, so that the results which matched your searches would be pushed to you, like tweets from your friends.

Today, there is Superfeedr’s track. It’s a simple way to get all the Atom entries matching a given keyword pushed to your app.

Once again, we hide all the complexity from you, and you can use the exact same API calls that you’re used too, whether it’s with the XMPP PubSub endpoint or the PubSubHubbub protocol. Are you tracking music as a keyword? subscribe to http://superfeedr.com/track/music. And yes, it works with any keyword like this : http://superfeedr.com/track/<keyword>.

Looking for a few apps that were built in less than 24 hours with Track? Check out this friendfeed page : it shows the track feed for the keyword Music. RSSFall by @jalada allows you to specify the keywords you want to track! Try out soccer, worldcup, iphone.. etc :) Want the results into your IM client? Check Harper’s SuperTrackr, just add @supertrackr@appspot.com@ to your contacts if you use gtalk, then, enter /track music and start dancing! Finally, @matijarijavec even built a mashup with Twitter’s trending words, pretty awesome for a few hours of work!

The results are taken from the more than 2M feeds that Superfeedr filters in realtime, as soon as the entries are published. These feeds include the feeds from the publishers who designated Superfeedr as their PubSubHubbub hub, like Six Apart, Tumblr, Posterous, Gowalla, Huffington Post, Gawker Media and more than 800 other big and small publishers, as well as the data from thousands of subscribers who offloaded their feed polling to Superfeedr.

We want more feeds! If you’re a publisher, start using us a a hub, and if your app polls feeds, stop doing it, it’s dangerous, have us push them to you.

Today we’re also launching our paid service for developers subscribing to feeds via Superfeedr’s default hub. Everyone will receive 25,000 free notifications to get started when developing with our infrastructure. We’ve come a long way in the past few months and while we’re still not perfect plan to make it even better with this support. Additional notifications start at only $0.0002 per entry which we guarantee will be cheaper than building it yourself!

One last thing!
As you’ve probably seen, the website’s design is changing… at least now it’s done by someone whose work is to design websites! Please bear with us as we put a little more paint here and there, but again, it’s moving in the right direction.

As always, your feedback is much much appreciated!

State of the Realtime Web : the Publishers

2010-06-22T00:00:00-07:00

The realtime web is almost as hyped a word as geolocation. It actually groups a lot of different realities and doesn’t have the same meaning and implementation from service A to service B. Let’s first focus on publishers.

The first thing we should all acknowledge is that every publisher can and should be part of the realtime web movement. We’d all be in a better position if we stopped assuming that Realtime web is the same as Twitter ecosystem.

Definitions

These are obviously subject to debate, let me know if you disagree in the comments, but I’m posting them first so you can understand the rest of the post.

User : you, me, them, everybody.

Application : a piece of software on the internet, that can be used by users, but also by other applications.

Content : a piece of data available on the internet. A blog post, a job offer, a product description… etc. Some people call that a resource. This brings us to the fact that most content should have a URL (Universal Resource Location).

Publisher : an application that publishes content on the internet. In more words : an application that makes pieces of content available at a given URL.

Pushing : an action by a publisher which consist of actually distributing the published content. Distribution involves sending the content away.

Pinging : an action by a publisher which consist of telling to some other application that some content is available. It doesn’t send the content, it just tells that the content is available for pickup.

The Realtime web Publishers

A small minority (yep, the opposite of a vast majority) of the content is currently being published in realtime. A lot of publishers will just make their content available without caring about the fact that this content is distributed. For a few years a few publishers where pinging another (smaller) set of people when they updated their content. You might be familiar with services like Ping-O-matic.
It’s not until very recently (maybe 18 months) that some services started to push their content away and distribute it. The main motivation behind this was to stop being polled over and over again for the same content. The first company who did this was Twitter, and their infamous firehose.

Twitter

Twitter opened the pandora box. By pushing their content, they allowed a whole new breed of applications which would consume the firehose in realtime and show it to their users. I know several people who were consuming Twitter only thru Friendfeed interface. At the same time, this proved like a very important monetization scheme for Twitter : search engines like Google have a very hard time crawling the whole web, yet alone high frequency publishing services like Twitter, so they paid a lot to get access to this firehose.
Yet, Twitter failed at establishing an open and standard protocol that could be used by other publishers to represent and distribute their data in a similar fashion. Worse, if you want Twitter’s data in realtime, you need to have a specific business relationship, which, in a lot of ways is the opposite to being open.
Their main motivation is still very present, and their ~~Chirp~~ User Stream is just another iteration which aims at making desktop clients like Seesmic or Tweetdeck much more efficient by pushing (in a proprietary manner again) them all the data for a given user.

Facebook

Facebook jumped on the boat very recently. Their “private” approach has long prevented them to push their content, because it would mean that they don’t have any control on the data once it’s out the door. Until a few weeks ago, they were fighting this by forcing any person who got content from Facebook to delete it after 24 hours. In practice, it meant that any service would have to poll Facebook every 24 hours for all the data they care about. Broken.

This changed a few weeks ago, because Facebook started to push some content (obviously people who hate their boss don’t get it). Facebook is becoming very good at communicating stuff, they hired a bunch of Open Web advocates and they now claim that they are open. Listen to me : Facebook is Open. Facebook is Open, Facebook is Open. Got it? Facebook is Open. Except that it’s not.

Of course, they say they use something similar to PubSubHubbub. Except that hum, the point of PubSubHubbub is to be an open and standard protocol. The standard part is very important. It means that services could interact together, since they speak the same language. Currently, this is not true : the PubSubHubbub from Facebook is not the PubSubHubbub defined by the spec, for many good? reasons, but calling it PubSubHubbub has more marketing value, I guess.

The worst thing about all this is that Facebook does push a hell of a lot of data… to whom you may ask? To Google. You see the pattern : an open platform that pushes content to some people (just not you, because your too small), based on business relationships, again.

The rest of the world of realtime publishers

By reading the 2 previous paragraphs, you may think I’m some kind of socialist and I hate business relationships. Well, I’m french, so that could explain it, but I’d argue that business relationships are also walled-gardens. Anytime you have a one-to-one, it’s about the 2, not about the others, right? If you married your spouse, it’s not to have ~~sex~~ dinner every night with a different person, is it?

In many ways, the relationships that Twitter and Facebook have with a very small subset of other application is de-facto excluding the rest of the world, and the rest of the future world.

Interestingly enough both Twitter and Facebook hide behind their APIs. Having an API is great, but it’s just another form of slavery (thanks @brad for the analogy). The consumers of these APIs are at the mercy of the API provider. Twitter decides to shut down HTTP Basic Auth? Sorry little app maker, you’ll have to shut down your app or abide by this new commandment “Thou shall not ask for user credentials”¹. Facebook changes their TOS? Sorry Zynga, no more invites for you.

If we want to build the realtime web, like we built emails or even HTML sites, we need a protocol. We need a way for services to interact safely with one another as peers, not as client and servers.

Luckily we have this protocol. We even have several (that’s the beauty of it : people fight for the shared wealth!). At Superfeedr, we placed our bets on PubSubHubbub and XMPP PubSub.

Most of the smaller (but much numerous) publishers have chosen the protocol approach. Whether it’s because they genuinely believe that it’s the good approach, or because they’re actually too small to force their API down other people’s throats is irrelevant. What matters is that now, all the main blogging platforms : Wordpress, SixApart, Blogger, Tumblr, Posterous are using the same protocol : PubSubHubbub. Some other smaller social networks, like Cliqset or Status.net and Gowalla are using it as well. Even a few newspapers!

Combined, this is probably bigger than Twitter or Facebook’s data. Soon, it will be much much bigger, because so many publishers don’t push their content yet…

The non realtime publishers

This is the last category of site, and still the vast majority. When looking at Alexa’s top 50, it’s obvious. Yahoo!, Windows Live, Baidu, Wikipedia, Amazon, Ebay, LinkedIn, Flickr, Craigslist, RapidShare…

Where are the e-commerce website sharing their catalogs in realtime with the price comparison search engines?

Where are the classifieds website pushing their content in realtime to iPhones?

Where are the Sports site pushing their content in realtime to forums or chat services?

Where are the news outlets pushing data to the feed readers?

One could argue that pushing data is letting 3rd party application use it. That’s my point actually : pushing data away is at worse pushing it to services with users, which means that your content will eventually gain eyeballs. At best, nobody cares about your content, so you’re safe.

Of course, we talked about the hundreds of new usages that are yet to be seen, from sync, to mobile, from presence to notifications… we haven’t seen anything yet. Please, publishers, let others benefit from your data. Publishing your awesome content without distributing it is pretty much like making the best product in the world but leaving it in the factory.

It’s time for content publishers to make their content dynamic and push it so they can control its distribution.

¹ I’m not advocating for HTTP Basic Auth, right? Just saying that thousands of small apps will shut their door, just because Twitter decided so.

Jyri Engestrom joins as an Advisor

2010-06-14T00:00:00-07:00

Despite the huge size of the web industry, the inside circle is like a small family. One of the great parts about this is that there is always somebody, be it another (successful) entrepreneur, a great hacker or an experienced manager who can help you on issues that you face daily as an entrepreneur.

Today, we’re welcoming Jyri Engeström to our Board of Advisors. He’s our third advisor, along with David Recordon and Andrew Anker. You probably know Jyri from his Jaiku days.

He’s one of the happy few who sold their company, to Google in the US, and managed social and mobile products at Google in Mountain View. He was born in Europe, created a few startups and ran products at Nokia, so he is very familiar with both the US and European ecosystems.

Jyri’s experience with this game is tremendous. He already helped Superfeedr on a lot of topics, and I learn so much when he gives advice that I feel I’m the dumbest person ever :)

His sweet spot is definitely all things communication. He’s helping me tune the Superfeedr message and worked through various presentations I made. Cool new improvements to our service are in the pipeline, so keep an eye out!

All three advisors have their own areas of expertise. Building a start-up is fun, and bringing talents to help in is definitely one of the most interesting pieces of this whole process! We’re now looking forward to bring more awesome people in the Superfeedr cave, whether they are employees, partners or just interns¹! If you feel that you can help, please get in touch.

¹ Looking for a summer internship? Loving GAE, ActivityStreams, Websockets, PubSubHubbub and/or XMPP? Get in touch, we can help each other!

A social filtering algorithm

2010-06-06T00:00:00-07:00

This post is not directly related to what Superfeedr is building but to what we’re seeing as a trend in the social applications field. About 10 years ago, a small company called Google wiped out any competing search engine with their PageRank algorithm. This one, without going into a lot of details was based on reputation of a web page : the more other pages have links to it, the more accurate it has to be.

If you’re a Facebook user (and haven’t quit yet :D), you’ve probably seen at the top of your News Feed, that you can choose between Top News and Most Recent. The latter is an exhaustive list of what all activities in your social graph, while the 1st one is only a subset of it. I believe the algorithm that is being used here is probably one of the few social filtering algorithm out there.

Why does it matter?

It’s no scoop to say that we’re slowly getting overwhelmed with data and activities in our networks. At the same time, this is data that we want to get, but data that we can’t deal with. This is what is extremely frustrating. I think it’s a tremendous value proposition for a service to provide me with the social data from my network in a way that I can consume in minutes, rather than hours.

Obviously this is very hard to build and even though I found Facebook algorithm ok, I keep clicking on Recent news to see the exhaustive news. I have also curated myself a lot of content (mainly by hiding apps, or fan pages that are too “verbose”). It’s probably harder to build because it’s a negative algorithm : it has to remove stuff, while the page rank and other algorithm are positive : they extract the relevant data.

If you’re doing any research on this topic, I’m sure you’ll find a job quite soon at any of the startups are trying to build out these social consuming applications.

Make your application real-time with PubSubHubbub

2010-06-03T00:00:00-07:00

~~Earlier in May~~ Last month, I was given the opportunity by my friend Brett to present PubSubHubbub at Google IO on stage with him. It was a great experience, and if you want to learn a bit more about PubSubHubbub and Superfeedr, I’d suggest you take the time to look at it!

This session will go over how to add support for the PubSubHubbub protocol to your website. You’ll learn how to turn Atom and RSS feeds into real-time streams. We’ll go over how to consume real-time data streams and how to make your website reactive to what’s happening on the web right now.

Social Real Time Apps Pub Sub Hubbub
These slides can be downloaded

Any comments welcome. Also, if you’d like me to talk at your conference or in your company, feel free to get in touch, it will help me get better at this!