Superfeedr Blog : Real-time cloudy thoughts from a super-hero

Chasing Real-Time Raindrops in an Ocean of Content

2010-03-11T00:00:00-08:00

The web is huge. And growing. Faster everyday. It’s almost like an ocean where there would be no evaporation (the data on the web stays there virtually forever), but yet, it’s always raining in it. The rain is the new content that’s added into the ocean. Every tweet is a drop, every blog post is a drop, every checkin is a drop that falls into the Ocean. This ocean is almost constantly under a tropical storm in some places, like Twitter or Facebook.

I’ve been invited by Marshall Kirkpatrick to write a guest post on ReadWriteWeb about the fact that Google will soon include PubSubHubbub data in its search engine. I hope you can read it and leave some comments there.

Ping me I'm famous

2010-03-05T00:00:00-08:00

Ping is both the ancestor and the root of any realtime feed protocol. A ping is a way to say : “hey this content was updated”. It can be light, which means the this gives a link to the content, or fat, when the ping actually contains the content.

As always with the internets, there are many ways to do that same thing. A pretty famous one is XML-RPC. PubsubHubbub has its own way of taking them into account too, RSSCloud as well… etc. They all work with Superfeedr, as we have implemented them all :).
By the way, if you want to start pinging us when you update your content, and if you already ping services like Pingomatic, Technorati or Google Blog search, you can also add http://ping.superfeedr.com/rpc to the list of services to ping.

Building a Ping Cache.

Most of ping protocols are actually HTTP Post requests sent to a url. That’s pretty easy to implement. Once we get a ping, we either extract the right params, or parse the XML body. That’s what’s after which is interesting!

Remember, pings can sometimes concern a feed url, but they can also concern a regular web page. Lucky for us, these pages often hold a feed, so we can consider that if we get a ping for a given page, we can consider that the feed itself as been updated. As a matter of fact, we use Redis (again!) to store a massive dictionary like this :

url1 => [feed_id1, feed_id2], url2 => [feed_id3], url3 => [feed_id1, feed_id2]

Each url might have several feeds associated, so we use an set. So far this is quite common. To be even more efficient, for each feed that we monitor, we fetch all the related “alternate” urls, and their “alternate” urls. What’s the purpose of that? Identifying clusters of urls that are updated at the same time.

Let’s take an example :

Let’s say a user is monitoring this feed http://superfeedr.typepad.com/blog/atom.xml. In our system it has an unique id of 123.

When we fetch this feed, to build the Redis-based ping cache, we see that it links (with rel="alternate") to http://superfeedr.typepad.com/. Hence, we’ll map both .../atom.xml and http://superfeedr.typepad.com/ to the 123 in our cache, so that if we receive a ping for http://superfeedr.typepad.com/ we’ll assume that the feed’s content has been updated. We also fetch http://superfeedr.typepad.com/, and this time, we see that it contains links (again, with rel="alternate") to :

http://superfeedr.typepad.com/blog/atom.xml
http://superfeedr.typepad.com/blog/index.rdf
http://superfeedr.typepad.com/blog/rss.xml

Similarly, we can now add to our cache the 2 extra feeds. As soon as we get a ping for any url, we will then assume that all of them have been updated!

Fetching the feed.

We have a complex scheduling mechanism. Contrary to the Google Demo hub, Superfeedr has an aggressive polling strategy for which we roughly poll any feed we have under 15 minutes. This amount of feed does that at every single second we have several thousands of feeds to fetch.

Our scheduling algorithm works in a non-deterministic way : it picks feeds that ~~will soon~~ need to be fetched, but on a random order. Obviously we can’t treat pings (for which the chance of update is hight) the same way as feeds that are just scheduled to be fetched (for which the chance of new entry is low).

For the pings, we use a deterministic system: a queue system where what goes in first, goes out first. Each of our dispatcher, are subscribed to the queue, and as soon as they get a message from it, they push it to some parsers. For the queue, we use RabbitMQ, and the excellent AMQP EventMachine library does an awesome job :)

It’s funny to see that our dispatchers are now multi-modal components: they are XMPP Components, send all of our HTTP notifications, and get messages from an AMQP queue. Each protocol is used at what it is best :)

A Hub for Twitter

2010-03-03T00:00:00-08:00

If you claim to be some kind of social app but don’t use PubSubHubbub, I think you’re missing out the opportunity to support a de-facto standard for service to service communication.

On the other hand, if you’re a realtime feed standard that is not used by Twitter, you’re also missing something.

Based on these simple facts, I think Twitter needs to have a PubSubHubbub hub. They recently announced that more and more apps are getting their firehose, this is great and shows us that they want their data out (hey, Facebook, did you hear that?). However, a few conversations with both John and Ryan showed me that they weren’t so keen of doing so. Sad.

To prove them wrong, we started a little poll 2 days ago. The goal was to show Twitter that people would use this hub, and that, with it, the consumption of Twitter data will eventually grow over time.

Here are the results (obviously, it’s never too late to answer) :

Would you use a Twitter PubSubHubbub hub if it was available?
Yes 45 96%
No 2 4%

I think this is the most important question here :) One could argue that the people who responded are also more tempted to say they would use it, than not. A legit question that somebody raised is that it would probably not work behind the firewall. I agree with that, but I also think that most apps who are behind firewalls use the regular REST API.

Do you currently use the Twitter Streaming API?
Yes 21 45%
No 26 55%

This is interesting to compare with the previous question. Almost everybody said they would use a PubSubHubbub hub, but less than half currently use the streaming API. This means that the streaming API is not fulfilling their needs, while a hub would.

Do you think a hub would increase your Twitter data consumption?
Yes 45 96%
No 2 4%

Again, that, I think, proves that a hub would be better suited for the data consumption for many applications. Also, I think that Twitter genuinely wants the data to “get out”. This proves that this goal would be achieved.

Have you already implemented PubSubHubbub?
Yes 30 64%
No 17 36%

A key concern when implementing an API is the speed of adoption. The streaming API has been out for a few months, but is still not so widely used, if Twitter had a hub, a lot of people would be able to start using it from day 1, because they already implemented the protocol on their ends.

I will send the detail results to both John and Ryan, as well as the list of people who responded (in a BCC email), so anybody can start the conversation from there.

Twitter will make the decision on whether they want to support that or not. It’s obvious that if they do, I’d rather see that happen via a Superfeedr hub. However, I think it’s still great for PubSubHubbub and the internets in general if Twitter decides to go down this route, but without us.

Like always, I’m here to help anyway.

Do You Prefer Buffet or Sit-Down?

2010-02-24T00:00:00-08:00

Editor’s note: As we mentioned a few weeks ago we want to offer our readers the ability to also write on that blog. This post by Doug Van De Motter explains why they chose PubSubHubbub and Superfeedr for Eavesdropper .

In the real time web world, currently there are two main choices for feed consumers – RSSCloud and Pubsubhubbub (PuSH). You can liken RSSCloud to a buffet for feeds and PuSH to a sit-down dinner. The hubs processing the feeds are the waiters taking your order – “Do you prefer buffet or sit-down?”

At the RSSCloud buffet, the waiter will take your feed subscription order, and when the cook (the feed publisher) has an update ready the waiter will be notified. The waiter in turn will send you a simple ping alerting you of the update so you can go to the cook’s buffet table and grab it. The waiter will also send a notice to all the other subscribers who will rush to the buffet table to get their updates, possibly crowding you out and making you wait. It’s also possible that the publisher may not anticipate the demand for the update and the table is overwhelmed with impatient buffet patrons who overturn the table.

If you prefer the PuSH sit-down dinner, the waiter will take your feed subscription order, just like at the buffet, but when the cook’s update is ready the complete order is delivered directly to you by the waiter using a fat ping, and you’re ready to consume as you like.

Being basically lazy and preferring our pings with a lot of fat, we opted for the PuSH sit-down protocol which we integrated into our Eavesdropper product using Zend Feed Pubsubhubbub and Superfeedr . The Zend Feed Pubsuhubbub component is new to Zend Framework 1.10 and makes integrating the PuSH protocol into an app very easy. We chose the Superfeedr hub for it’s standardization into Atom of any feed to which we want to subscribe, and because they handle the complexity of the alphabet soup of feed standards so we can just relax and enjoy our meal.

Eavesdropper is our real-time monitoring tool that can be used for reputation management, competitive intelligence, lead generation and crisis management, or many other uses depending on your needs. With the addition of PuSH, you can monitor any feed of interest en – dash for example, you may come across a blog post through Eavesdropper where users are actively discussing your product or service, and if the comments are feed-enabled you can subscribe to them directly within Eavesdropper to follow the discussion as it happens.

So, as a hungry feed consumer, do you prefer buffet or sit-down?

On Outsourcing

2010-02-23T00:00:00-08:00

I catch a small conversation between Alex Payne and Matt Galligan today :

Al3x : Something you hear over and over again from people running startups that made it: “[some capability] matters too much to outsource”.

Mg : @al3x I generally like to call bullshit on that statement

Needless to say that I fully agree with Matt. As a matter of fact, this is something that we hear a lot from people we go after. And I think this is terribly wrong, specially when you’re a small startup.

I think most people agree that the biggest hurdle for most startups out there is not proving the technology anymore, but proving the market. If you don’t agree with that, just read Dave McClure’s blog and presentations : doing tech stuff is the simple/easy part of doing a startup. Proving the market, getting users and eventually customers, milking their needs to make your own profit is the hard part.

If you have that in mind, when you start a company, the first thing you want to take out of the way is the technical stuff, you don’t want to spend even minutes building stuff that somebody else already built. As a matter of fact, if you start building stuff yourself, there is no chance that what you built in the early days is even going to last as much for a few months : why doing and re-doing something that somebody did already?

Also, what matters too much, when you have nothing? When I hear people building small apps saying “this feature matters too much”, I can’t help but think “matters to much to what?”. The only think you should preserve at all cost is your time! Luckily you can’t outsource that anyway.

When your startup grows, when you’ve proven the whole market thing, then, yes, you can start to decide what matters most, what you can do better (then the others), and what you want to take internally, rather than leave outside.

Also, the internet is now the biggest toolbox in the world, you can find services that do analytics, hosting, geolocation, realtime feeds, image hosting… and software for everything else : building a startup is all about picking up the right bricks, and stacking up with a good mortar. It’s not about making the bricks!

UrbanAirship

2010-02-22T00:00:00-08:00

We’ve had a few customer feedback and use-cases in the past. This week, I’d like to write about UrbanAirship. We’ve been in touch with Scott and Michael for a while now, and they recently announced a Series A round.

For this time, I have sent 5 questions over to Mike… here they are!

What’s your product/service?

Urban Airship provides an easy way to deliver in app content and send messages to mobile customers. We’ve delivered over 130 million messages and talk to over 11 million phones. Our key strategy is to make it incredibly easy and cheap to get going with in-app purchase or push notifications. Our simple RESTful API makes it a breeze to integrate into an existing system and we support a lot of advanced features, such as arbitrary tags, aliased users, or broadcast messages.

How do you use Superfeedr for?

We want to make it really easy to integrate push notifications to existing services. One easy way to integrate messages into an existing flow is to hook up an RSS/Atom feed. When that feeds gets updated, we wanted our customers to be able to send a broadcast message to all their users with those updates. This makes some applications very easy to build. So, now, developers can just hook up an RSS feed to their Urban Airship application and broadcast messages are delivered – no fuss required.

Why do you use Superfeedr? (vs building your own!)

We’re constantly adding features and working on improving our service. We’re also a very small team. We didn’t want to bother with building out yet another component of our infrastructure dedicated to this feature, especially when we know how tricky parsing a large number of potentially unsafe and poorly formatted feeds can be. That’s why Superfeedr was an obvious choice. It was a very easy integration and we can rest easily at night knowing that that’s one less thing that we have to worry about.

What’s your stack? Language? Superfeedr API you use?

We’re a Python shop – our front end is written in Django with custom back-end services for message delivery. We choose the PubSubHubbub API since it was the easiest to get up and running. It’s also much easier to maintain a Django application dedicated to it than another infrastructure component.

What would you change if you were in charge of Superfeedr for 1 day?

I think I’d give Julien a vacation – he’s constantly available for help and seems to be everywhere he can be lending a helping hand.

Redis at Superfeedr

2010-02-20T00:00:00-08:00

A lot of the problems we had to tackle in the last months were directly related to the data stores we used, as well as the schema of the objects we stored.

We’ve been using MySQL and Memcached from day one, because it’s always good when you have 50 different problems to tackle to use tools that you know (read this article by @mcuban if you’re about to take the red pill).

However, given our growth, we reached a step where the complexity of scaling horizontally involves sharding and other stuff that we weren’t convinced MySQL would do well (at least, because we need to have Memcached in front too). Rather than heading too far into bending MySQL, we thought we should give a chance to Redis. And here the story.

How it is/was

Let me start first to say that we’re not dumping MySQL + Memcache altogether. We identified one type of data for which it wasn’t appropriate anymore : the feed entries.

One of the key things the hub does is diff-ing : identify new entries from old entries in the feed.

We do this by storing some kind of data when for each entry, in each feed. We use a hash of the entry id in our schema. The problem is that a feed can have several dozen of entries, and we obviously need to keep track of them all, plus some older ones, because you don’t want entries to “re-appear” in a feed (if a more recent entry has been deleted). When you have say 1M feeds, and you need to store 50 entries per feed on average, that’s 50M entries. And every time we fetched a feed, we need to compare the existing entries to see if we’ve seen them already¹. And we fetched all our feeds at most every 15 minutes : that’s a lot of queries.

Also, some feeds have new entries every few seconds, if we leave that for a while, say, 2 days, that’s a lot of entries stored that we’ll never see again. So, as soon as we see a new entry in a feed, we have to clean older entries.

Needless to say that this whole traffic is too much for our “simple” MySQL setting on a 8GB of RAM server. We had to put a Memcached in front of that to store the most recent entries. We obviously can’t use Memcached alone, because if the server goes down (or if we just need to reboot, update the configuration it), all the traffic will end up on MySQL which will obviously die in hell in seconds.

Requirements

As a matter of fact, we have the same exact information in Memcache and MySQL. We can’t use either of them alone. MySQL is just too slow to deal with that much traffic (even with all the InnoDB indices in RAM), and Memcache isn’t persistent, so we can’t accept to use it alone. Also, we actually store our very own datastructure into Memcached, because it only deals with key/value that are strings. In our case, we have a key which is our internal feed id and the value is a string of all the entries’ id joined together.
Also, we do some sharding on Memcached (on the client side), based on the superfeedr-wide feed id, and we’d love to keep that.

We wanted to find a tool that has the “speed” of Memcached, some persistence, and which should be easy to shard on the client side. Also, we want to use a store that could just silently fail, to avoid any bottleneck (at the risk of having duplicate entries). Also, this tool should ideally have some kind of data-structures so we don’t have to hack one that can be mapped in strings at a low extra-size cost.

A few months ago, I heard about and played with Redis. I was quite impressed by it and thought I should keep that in mind. A few weeks ago, Antirez added 2 awesome features that made Redis an excellent candidate : Append-On-File persistence (instead of periodic snapshots) and Sorted Sets.

Implementation

We use ZSETS (sorted sets). Each feed has it’s own sorted set. The key is the global feed id, the members are the entries ids, and the scores are feed entries timestamps_. When fetching a feed, *we compare the uniqueids* in the set very efficiently (in parallel). Then, when the size of the sorted sets goes beyond an arbitrary limit, we use the score to delete the oldest entries. Easy!

For the client, since all of our components are evented, we use em-redis. It allows us to run the comparisons of new entries in parallel. However, we had to hack it a little bit to allow silent fail when/if the server is not present.

Obviously, we use AOF for persistence, so even if Redis goes down abruptly, we shouldn’t lose any data. We keep the regular snapshot (BGSAVE) every 15 minutes or so, just to make sure we have the data twice (and we copy it somewhere else).

We shard based on the feed id. We do not use a hash key, because (even if we used consistent hashing), we would have duplicate entry for each new redis server we add to our farm. We based that on the fact our ids are consecutive and we store 250k feeds per server. id < 150000, redis = server1; if id < 300000, redis = server2 .... that’s not the cleanest thing but that works :)

We use 2GB slices at slicehost and all the entries should fit in there, which is good. If not, we’ll keep less entries per feed (we currently keep between twice and 3 times as much entries that there are currently in the feed.)

When getting into prod, we had a few issues with Redis and the performance was very disappointing, specifically at the time of the snapshots. It appeared that it was degraded by some changes antirez introduced in the last version. We were delighted that he acknowledged the issue very fast and even used one of our boxes to debug. A few hours later 1.2.2 was out, fixing all the issues! Who said customer service sucked with Open Source Software?

Tip

If you can run redis servers with less than 4GB of RAM, compile² and run it in 32bits : redis is very greedy in terms of pointers and 64bits pointers are twice as long as 32bits pointers (suprising!). On one of our servers, we gained 40% (from 1602428 to 1002124). Pretty big! Also, you want to have servers with less than 4GB : it’s probably safer to have many small redis instances than just a few big ones.

Conclusion

It’s been now in production a few days, each of our redis servers process on average 3500 queries per second. They store on average slightly less than 1GB for 200k sorted sets (with about 4k per key/value), which should give enough space for more entries. The load is consistently below 0.1 on a quad-core. This will probably not grow much, so we’re more RAM bound than CPU bound here, so you want to implement the trick above :)

It feels good to not have an “horizon” anymore in terms of scale for the entries. Our only concern is to find the best way to provision and deploy more redis servers when we need them :D

¹ I’m oversimplifying, we use Etag, If-Mofidied-Since and a global feed hash, to avoid comparing that every times, but that’s still pretty high!

² Change the 32bit target: to make ARCH="-m32". Then, make 32bit

Google's Buzz is PubSubHubbub in, and out

2010-02-09T00:00:00-08:00

Today, Google launched Buzz. Aside from the great social features that it brings, I want to emphasize how PubSubHubbub is at the core of it. As Brad (who is clearly behind that in some way) said, Buzz is PubSubHubbub both in, and out. This means that Buzz gets its data via PubSubHubbub, but also publishes it in realtime, via PubSubHubbub as well.

From day 1, Google actively promoted the PubSubHubbub protocol on the publisher side, but they took a while to integrate the subscriber side that in existing products, like Google Reader, iGoogle, Feedburner… etc. Today not only they did that, but they also put it at the core of a brand new product.

In the past few weeks, we started to see a ton of subscriptions from Google to a lot of feeds for which we host the hub. I’m pretty sure that soon all Google properties will use PubSubHubbub for the feeds!

Huh, and the rest of us?

It simply means that if you have a Posterous Blog, a Tumblr Blog, or if you have a feed for which superfeedr host the hub, you can (and should!) add that to your Google Profile so that the next story you publish is available into Buzz almost instantly and your friends can see, comment and like it :)

Obviously, PubSubHubbub (between services) and XMPP (behind the firewall) will be the base layers for the realtime and social web. Whatever your app is, you need to make it realtime by supporting PubSubHubbub in your feeds!

PubSubHubbub : security

2010-02-08T00:00:00-08:00

One of the great advantages of our XMPP API is that there is no need for the subscriber to know whether the content he gets is actually from Superfeedr. PubSubHubbub doesn’t benefit from that, since it relies on HTTP callbacks which makes it hard to know where a notification comes from. A subscriber could obviously check the IP as well as the UserAgent, but the first one will break when we add servers and the second one is pretty easy to spoof.

The protocol defines a strategy for that and we implement it at Superfeedr. Here is how it works :

1. The user subscribes to the hub url and provides a hub.secret value. To make that secure, the subscriber should use the http*s* url to the hub. (All our hubs have an http and https url).

2. Upon notification, the hub will compute an HMAC signature with the supplied hub.secret, as well as will the body of the notification, and will include this signature in the headers of its notification.

3. When notified, the subscriber will compute the same signature (he should have kept track of the secret!), and will compare with the http header received from the hub.

If the 2 HMAC signature are the same, then, the subscriber knows for sure that the notification comes from the hub. If not, it should silently ignore it.

Gawker Media and ReadWriteWeb have hubs

2010-01-29T00:00:00-08:00

ReadWriteWeb is one of the most respected tech blogs in the Valley. Marshall, Frederic and Jared Smith, along with Erick and Steve at Techcrunch, have been early realtime supporters. ReadWriteWeb also wrote a very insightful report on this trend and you should read it.

Gawker Media is one of the biggest and most successful blog-oriented media companies. Their publications include the following names : Gawker.com, Defamer, Fleshbot, Deadspin, Lifehacker, Gizmodo, io9, Kotaku, Jalopnik and Jezebel.

Today, we’re quite proud to say, that they now push their content in realtime : they have their hubs : http://rww.superfeedr.com, http://gawker.superfeedr.com.

We’re very happy to have gawker and rww using Superfeedr. In the past we announced hubs for blog platforms, and we’re excited to see what will happen for another kind of publisher : media companies.

It’s been amazing to see how some implementers of the PubSubHubbub protocol already found their hubs and subscribed to several feeds¹ :)

You (and your app) could also benefit from a superfeedr hosted hub if you do publish content on the web. Reach out!

¹ To be fair… they almost forced us to write about these hubs!

10 Feed Publishing Best Practices

2010-01-27T00:00:00-08:00

Working with publishers and polling hundreds of thousands of feed has taught us a few lessons and best practices for people who publish RSS or Atom feeds. You might disagree with some of them, so please, let us know in the comments.

I want to emphasize that these best practices are technical, not business related.

Don’t use RSS and ATOM. There is no good feed parser that doesn’t accept both. Providing one will make subscribers life much easier :)
Make sure your feeds are valid.
Use unique identifiers for entries : <guid> in RSS and <id> in ATOM
Don’t use feed caching if you can’t explicitly expire the cache. In other words, don’t use time based caches. Corollary : use Feedburner only if you use pingshot.
Make sure your date format is valid : ISO8601 is the one you should choose.
Add the feeds discovery in your HTML pages. Most modern browsers (Chrome : WTF?!) will be able to detect that and show your visitors that the page references a feed. It will also help feed readers to auto-extract feeds.
Make sure you’re RESTful in your feeds discovery. If you’re in a resource page, show feeds that relate to that resource. Make sure a given feed is accessible at a given url only.
Use the “de-facto” standard in terms of calendar : Roman. The subscribing app may convert to other calendar, based on the user’s settings.
Keep the feed urls simple. Do some URL rewriting.
It’s ok to have only <summary> instead of the full <content> (it’s a business decision) : but don’t show it as full content if it’s just a summary.
Special Bonus : publish feeds in realtime. We can help.

Thoughts?

State of Real-Time feeds

2010-01-26T00:00:00-08:00

It’s this time again : the time for the very controversial numbers about realtime feeds. Before anything else, I’d like to indicate that it would be great for other services to publish similar numbers. We know who implemented PubSubHubbub, I guess some of them implemented RSSCloud and SUP too.

Since we have so much debate in the past about the methodology:

Superfeedr has many feeds in management. Some of them are for the hubs we host. We don’t count the feeds from the hubs we host, because they would obviously inflate the PubSubHubbub number a lot.
For each feed, since we try to avoid polling it too, we use any available technique to get in realtime, se we’re able to tell whether it’s PubSubHubbub, RSSCloud, or SUP. We just counted these.
Some feeds implemented 2 or more protocols (the best example would be wordpress.com blog that have a feedburner feed), we counted them in both categories.

A few notes on the result set before. We have multiplied by 6 the number of feeds we managed since the last time we count, which should make our set a little more representative. As a matter of fact we gained a lot of feeds that are not blogs.

	Polling	RSSCloud	PubSubHubbub	SUP
Oct 1, 2009	92.7%	0.3%	7.1%	0.0%¹
Nov 1, 2009	73.3%	0.8%	25.8%	0.1%
Jan 26, 2010	87.4%	0.9%	11.5%	0.1%

You can immediately see that the proportion of feeds we’re polling increased compared to the last measure. This is the consequence of less PubSubHubbub feeds. Back in november, we counted the PubSubHubbub feeds we were hosting. That was a mistake, we changed that this time and will not count them in the future either. The RSSCloud proportion grew in comparison, since PubSubHubbub numbers were inflated.

We are now continuously measuring the proportions, which should make these posts way more regular. I look forward to the next month!

In conclusion, I’d like to remind you that we implemented many other ways of getting feed updates, including some specific APIs, XML-RPC Ping and other techniques. If you use Superfeedr, you will benefit from these and get feeds in realtime!

¹ At that time we had not implemented SUP.

Why realtime matters for publishers

2010-01-25T00:00:00-08:00

Several people wonder if realtime feeds is something they need, since they have regular RSS and Atom feeds.

Now

The web now is organized in such a way that every service or person needs to go fetch any information, or have to “accidentally” find it again. No matter how great your content is, you’ll have to wait for visitors to come, and once they’re gone, you have very little way of telling them about new content.

I was surprised to find out last week, during the S2S diner in Palo Alto that the biggest email sender were social networks (except for spammers, obviously). They send you an email for a LOT of things : a new friend request, somebody tagged you, somebody mentioned you… etc. This is their way of pulling you back to the site.

When you look around, the sites which generate the more engagement are the sites who send the best emails, because email alerts is almost the only product that can get you back to a site regularly.

Unfortunately not all services can rely on email. As much as giving personal information (like your email) to a social network is ok, it doesn’t make sense to subscribe to any mailing list out there. Also, when a social network sends an email, it’s always accurate and personalized! If I subscribe to the NYT’s mailing lists, there is actually little chances that the emails I am going to get will be relevant.

This situation is also giving a huge importance to search engines. If you have no way of reclaiming your visitors, you want to always be in their way when they search for content. Yet, there is no guarantee that your rankings in Google or Bing will stay identical forever. On top of that search engines are not the best way to work on your branding: from a google search result page, can you tell the different between a NYT article and a page from somebody that can’t really be trusted? No.

In realtime

The realtime web is synonymous with pushed web. I even said it in the past : the time factor doesn’t matter much. What truly matters is that the information is pushed, that it will eventually reach services/people away even if they do not come to check it out.

In the process, when a publisher pushes its content, he is the “initiator”, he is active. When a publishers waits for others to come check its content, he is passive and relies on the capacity (and willing!) of others to come by themselves.

Becoming a realtime publisher means being in control, rather than relying on others.

Eventually, when your service will build its traffic, both approaches will exist :

Passive: you update your content and leave it as is, ready for people to randomly find it (through search engines, link sharing… etc)
Active : you push it to subscribing application : feed readers, content aggregators, feed to email gateways, feed to SMS gateways, realtime search engines… etc

By building hubs, we make it very easy for any publisher to publish in realtime. You need one, get one now!

A hub for Dotspots

2010-01-24T00:00:00-08:00

As Matt announced earlier this week, Dotspots now as a PubSubHubbub hub, hosted by Superfeedr.

DotSpots enables anyone to update the news in real-time with dots: mini-blog posts, including videos, pictures, documents and links, that appear in context on all relevant news articles.

Using superfeedr’s hub is quite easy and involves 2 steps :

Adding the discovery to the RSS/atom feeds. This is just a small XML snippet to indicate that a given feed has a hub and its location.
Pinging superfeedr. Even though we always failover to polling, it’s also always better if you can ping us when a feed has been updated.

I asked Matt some feedback on the process of using Superfeedr’s hub, and he kindly sent me the following :

Turning on real-time updates with PubSubHubbub was really easy. We use the Rome library to output Atom feeds for a number of different views on the data. We added a simple hub link to each feed to do this:

Our architecture already includes an asynchronous queue of events for mutation of different objects. We just needed to add another listener on the events for dot creation and modification that notifies the Superfeedr hub of changes. When a dot is modified, the event queue notifies the PubSubHubbub notifier, which then pushes out the updates to any of the feeds modified by the action. Notifications use the built-in Java support for HTTP:

As you can see, it’s quite straightforward… You can start today : Setup a PubSubHubbub hub

Compliance for Superfeedr's Hubs

2010-01-23T00:00:00-08:00

We said it in the past, abiding by standards makes the web better. Yet, sometimes, standards have a tendency to diverge from an implementation to another. You know, the theory vs. practice thing.

PubSubHubbub is a standard. Since it’s relies on HTTP, it makes it quite easy to test and specify, and that’s what progrium did a while ago.

Today I checked and cleaned a little bit the test suite, and I uploaded it to Superfeedr’s github repo. I tested it with the hubs we host and, good news : we’re fully compliant on the subscriber part :)

The publisher part is a little bit trickier, because it expects the hub to fetch a url after a ping. However, in Superfeedr’s case, the scheduling algorithm is a slightly more complex, and it makes it “non-deterministic” from the outside world : quite hard to specify.

Obviously, you can run that code by yourself to make sure we’re doing good! We will also try to update both the hub and the test suite when the specification evolves! (Also, I need to find a way to push that back to Google’s svn, but my svn is probably not better than my ancient greek…)

Who Implemented PubSubHubbub

2010-01-21T00:00:00-08:00

Exactly like last month, here is a list of domains for which we have callback urls : they implemented PubSubHubbub!

Let’s start with the new kids in the room, who were not in the list last month, again, in absolutely no particular order:
- netvibes
- tailsweep.com
- rss2twitter
- fliptop-inc.com
- geowhy.org
- sfr.fr
- viewsflow.com
- chitthajagat.in
- tbablogs.com
- webwag.com
- nababu.org
- pitchfork.com

Obviously, all the big guys from last time are still here :)

In total, we now have 177 different domains with callback urls. This is very exciting!

Publisher Callback Extension to PubSubHubbub

2010-01-20T00:00:00-08:00

Superfeedr now hosts several hubs. We had discussion with a lot of data publishers, including blog platforms, but also social network and regular “media” sites.

All of the people with whom we talked were excited by real-time feeds and most of them see PubSubHubbub as a good way for them to rely less on polling and actively push their data toward subscribers. However, a lot of them raised the concern that the protocol was a little too blind for them. They want to have a certain visibility on who (the callback urls) subscribed to what (the feed urls).

The concern goes from regular logging and backup of the subscriptions, to spam control or even paid-subscription.

The following scenario scared a lot of the media sites we talked to: we all know that a lot of spam sites crawl and ‘steal’ content from them. They put ~~a lot of~~ ads and usually use black hat SEO to obtain better rankings in the search engines. PubSubHubbub makes the spammer’s life easier because they don’t even have to poll for the data anymore. There are several techniques for publishers to filter spammers when they poll (IP filtering… etc), but they don’t apply to PubSubHubbub since the publisher is blind.

Obviously this is a legal matter more than a technical matter, but there is no reason either to provide spammers with a tool which would make that easier for them (specially since it’s not always easy to enforce copyright internationally).

Another use case is to allow subscriptions from certain feeds only. Take our Tumblr hub as an example. We can only allow subscriptions for feeds that are on Tumblr. The callback mechanism can help us query Tumblr to make sure they actually host a given blog (when the blog uses a custom domain, it’s not always easy to tell).

We published the proposal for this (optional!) extension, we’d love to have feedback from the community. Obviously this is a draft, which means that it NEEDS more work :)

XML-RPC Ping to PubSubHubbub

2010-01-13T00:00:00-08:00

Slow blogging these days, but we’re still (more than ever!) crushing it :) XML-RPC ping is the grand father of all ping mechanisms for feeds and blogs. There are a few famous ping servers, including Google or Ping-o-matic from Automattic. Unfortunately, the lack of “subscription” have made this ping servers a nest for spammers. Check the changes.xml files to see how many notifications are spam vs. legit. Sad :(.

Yet, many publishers have implemented this protocol, and we decided that there was no reason to leave this people on side, so we mapped this protocol into PubSubHubbub so if people subscribe to your feed and you ping us, we will take that ping into account and send them the update in real-time!

If you’re a publisher and if you have implemented the ping protocol, then, the next step for you have a PubSubHubbub hub is just to add the discovery xml into your feeds and let me know so I can plug your feeds to our hub :) It shouldn’t take more than a few minutes on either side!

Scaling Superfeedr

2010-01-06T00:00:00-08:00

I’d like to start this post by wishing everybody a wonderful 2010 year, and a great 2nd decade of this millennium.

We will remember these early days are pretty hectic, here at Superfeedr. As a matter of facts our users keep feeding us with more and more feeds (+20% in a week!). It’s great, except that it makes little issues much bigger. And we’re currently fighting hard (like 16hours a day) to improve our scheduling algorithm.

The difficulty that we have is that we need to deal with a lot of pings, telling us that a given feed has been updated. Technically, at the moment when we receive a ping, we need to fetch the feed and push that to our subscribers, which means that we’re already late to deal with this feed.

So, at any time, we know how many feeds are late. Also, we need to fetch all of our feeds at least every 15 minutes, which roughly tells us how late we are on average for each feed.

Right now, we’re at about 10% of late feeds, which means that, on average, we’re 1.5 minutes late on pings. Yes, it’s much higher than acceptable and we’re working on that – but it reached 50% earlier today :( -

Contrary to many services out there, Superfeedr has to scale “internally”. Most of web apps and web services usually work like this :

A user sends a query
the application processes that query
the application returns the response.

It’s not always easy to scale all the time, but at least it’s a fairly known science. Adding cache to skip the 2nd step is the most obvious thing and it can do miracles :)

Unfortunately for us, that is not how Superfeedr works. Of course, like I mentioned we deal with incoming pings, but we also have to deal with polling for a lot of feeds (even the pinged feeds, as a failover), which means that as opposed to almost every web service out there, we do not act upon user input. The good news is that we can probably have a bigger query → response time. The bad news is that caching¹ is useless!

As a matter of fact, our biggest problem is to distribute all our data-processing smoothly and evenly across our whole infrastructure so that we always have a capacity utilization of 100%. And that’s what I’ll be spending the next hours and days on :)

¹ And caching is evil for real-time stuff :)

Feedoor and Superfeedr

2009-12-29T00:00:00-08:00

Earlier this month Mamod sent me a notice saying that they enabled real-time feeds management to any website or blog (See Arabcrunch) used through Feedoor.

Mamod was kind enough to create this great presentation on how he integrated Superfeedr into Feedoor :

Superfeedr Integration With Feedoor

View more presentations from mamod.

He also sent me the following :

Our major concern was to make all bloggers able to enable real-time support for their feeds, and Superfeedr helped us to achieve that.

In the beginning we supported RssCloud protocol and then Pubsubhubbub, these protocols are the most well known protocols in the world of real-time, however, the problem lies in the lack of real-time protocols in most feeds, which means that we need to find a smart solution to overcome this problem. The solution is to poll feeds with no real-time protocol periodically and intelligently, and then match with the previous content, if there is new content we ping feed services and subscribers, this is the only way we can achieve as close as possible to real-time for feeds with no real-time protocols. For those who do not know how difficult this process is,* the process of polling thousands of feeds several times a day needs lots of resources and planning, it’s simply a very crazy task*. Luckily we found a crazy service which already do that exact thing we needed and surprisingly they do it right and efficiently, Superfeedr, they’re so crazy but glad we found them. Now when ever we manage a feed with no real-time protocol we send it to Superfeedr and they do the polling and the checking on our behalf, and all we need to do is waiting for their pings :)

As for how to integrate Superfeedr in our service, we used their HTTP API, it was very easy and just like dealing with Pubsubhubbub real-time protocol as they use the same specifications, they are a big hub that supports all real-time protocols and smartly poll feeds frequently.

Here is what Feedoor used to integrate with Superfeedr :

Superfeedr API: HTTP API
Programming Language Used: Perl
Modules and Libraries: In house

Who Implemented PubSubHubbub

2009-12-28T00:00:00-08:00

PubSubHubbub is now more or less 6 months old. We have daily talks with people who wonder whether they should implement it in their feeds.
Obviously, it matters for publishers to know who will subscribe to their feeds if they use a hub for their feeds.

Our monthly list market numbers show a pretty interesting image in terms of how many feeds are PubSubHubub enabled. Here is some more “qualitative” data to assess who implemented the “subscriber” side of the protocol (get started with PubSubHubbub).

Since we now host several official (Posterous, Tumblr, Backtype, enjoysthings… more to come!) hubs and a few unofficial (identica, wordpress)… We can then know quite easily which services have subscribed to these hubs for some feeds published by these guys.

In no particular order, here are the main services for which we have callbacks :

We have about 200 different domains with registered callbacks, but we couldn’t really identify the service to which they relate. Also, these numbers do not take the subscriptions to Superfeedr’s hub into account. Just the hubs we’re hosting.

There may be a few other (please tell us in the comments), but we can’t confirm that they subscribed to any of the hubs we host though.

I tend to think that the number of services who subscribes matters more than the number of publishers from now on. Let me know your thoughts on that.

Distributed and Federated Social Networks

2009-12-21T00:00:00-08:00

There is a lot of debate around on OpenTwitter, OpenFriendFeed OpenFacebook. these days. The de-facto standard which is now Twitter’s API is a good news. Yet, social Networks have 2 big features : publish content (it seems that Twitter’s API is the reference now), and consume content from your friends. This second part seems much harder and I think PubSubHubbub and XMPP can provide this API.

Whatever the social network, user profiles usually have a link (Atom or RSS) which represents a user’s stream of activities. Whether it’s Flickr, Delicious, Twitter, or even your blog (yes, it’s a social network in a way), all our online activities are represented as feeds.

The next step is to make these feeds “real-time” so that the consuming applications shouldn’t need to poll thousands of feeds or services. To consume my friends information, I shouldn’t even need to be part of the network on which they publish. PubSubHubbub enables that.

I like consuming my friend’s feed via Facebook. Yet, a lot of my friend’s content is not there : their tweets, the pictures they post on Flickr, etc… and some of my friends are not even on Facebook.

I wish somebody (Tweetdeck, Seesmic, Brizzly… can you hear me?) would have a built-in PubSubHubbub subscriber. Then they would allow me to paste one of my friend’s profile page on Last.fm or Foursquare (or allow me to select directly pages on specific services). From that page, they would auto-discover the feed, and then its hub and subscribe to it. Finally, when my friends update their social network (provided that this one uses PubSubHubbub to make its feeds real-time, but we can help), I would receive instantly the update into my Social Network consuming app.

When publishing my content, I shouldn’t have to publish it on 36 different services just to make sure all my friends will get it. It should go wherever my friends will consume it!

Be my guest

2009-12-19T00:00:00-08:00

We use Jekyll as our blog engine, and we use github to host said blog. As a matter of fact, our blog is a regular git repository. This as several consequences, including the fact that it’s quite easy for us to add “remote” contributions. If you want to write about Superfeedr related stuff : your experience with Superfeedr, but also the technologies that we use (XMPP, PubSubHubbub, Chef, Collectd…), or even your thoughts on the real-time web; this space is yours.

It’s actually quite easy, you just have to fork our blog’s repository, write your own blog post in the _post directory.

The syntax is fairly simple : name a file yyyy-mm-dd-title-of-your-post.textile, then, add your post content. Make sure you follow the “Textile syntax”: if you want to make your text bold, italic, add links or even titles… Finally, add a top part to the file like that:

When you’re done, you should test the layout, by using the Jekyll :

Install the gem (if you have ruby installed)
Type jekyll --server in your fork of the blog’s source.

Finally, do a pull-request. If we like it (we will try to keep our censorship to a minimum level, but we need to make sure your posts aren’t too self-advertising and are still related to Superfeedr), we’ll publish it!.

Done! This place is now yours too :)

Feediscovery

2009-12-16T00:00:00-08:00

At Superfeedr, we believe we have to do one thing only in the best way it can be done: make feeds real-time.

There are a ton of small feature requests that our users ask for and that aren’t part of this grand scheme. One of the most common ones is feed discovery.

A lot of apps that are built using Superfeedr allow their users to type in any site url. Since Superfeedr only deals with feeds, it means that these apps have to

Download this specific page,
Parse the HTML
Extract the <link> tags to get the corresponding feed(s) urls.

It’s definitely not a very hard thing to do, yet, there is* absolutely no reason that everybody should write the same code over and over* for that basic purpose. Obviously, you could use great libraries such as feedbag, yet, these libraries are language-centric (Ruby in this case).

We built feediscovery : a simple – in the cloud! – feed discovery API. To use it, send a GET request to http://feediscovery.appspot.com/ with the url parameter from which you’d like to extract feeds from. The app should return a small json string of the feed(s) information (href, title, rel, type).

Example:

GET http://feediscovery.appspot.com/?url=http://blog.superfeedr.com
→ {"href":"http://blog.superfeedr.com/atom.xml","title":"Superfeedr' thoughts","rel":"alternate","type":"application/atom+xml"}

We truly believe web apps are soon to become the next base-component, and will, in many cases, replace libraries, gems, packages, modules.

How we built Analytics

2009-12-15T00:00:00-08:00

We weren’t quite sure how to build these analytics. We slowly established a set of requirements and constraints

Zero performance impact
Fully decoupled from the current infrastructure
Results at most hourly
Data is more important than graphs
Easily-extensible, in case we want to measure more things

Collecting

There are 2 ways to collect data : on the fly, when things happen, or by just collecting stuff in the database. Since the top requirement was to have 0 performance impact, the most obvious way was to have a “maintenance” script that runs forever and measures what needs to be measured in a loop.

We added a slave server to our main MySQL server(s). We use this server for the analytics. It’s a regular slave because, obviously doing analytics is about reads, not writes. With that, we can guarantee that the performance impact on the current is zero. If the analytics dies, nothing else will be impacted.

The scripts runs at regular intervals so that it takes several set of measures by doing queries on the slave database.

Storage

Since we need to have an extensible schema, we eliminated anything with a “fixed” schema, like MySQL. Additionally, analytics can really be regarded as documents, which naively lead us look at Document stores. We chose MongoDB. It’s probably the most actively maintained of its category and Ruby has a pretty neat library: MongoMapper.

It also has some MapReduce functionalities, which will probably be useful when start having a lot of data and we want to make complex analysis of it. Unfortunately, there is no event-machine driver for it, but it seems that inserts return immediately, which shouldn’t be too bad in terms of performance.

Querying

Right now we don’t have a lot of data, so the regular MongoMapper interface does a pretty good job. Typically, here is how we process results : we get all the data for a given time query in a given time-frame, like a day. We then so some kind of Map-Reduce to get results on an hourly basis : we split the result set by hours (map), compute averages/max/min (reduce) and then return the final result set.

When we’ll start computing more complex stuff, we’ll probably need to use Mongo’s capabilities for MapReduce, but right now we’re “emulating” that behavior from within a Ruby script.

Since the data is not expected to be updated more than hourly, we also cache the results in our Memcache server. Another option that we’re evaluating is storing this as actual documents, so we have the raw data (collected from MySQL), as well as the computed results.

All the results are returned in a Json format, which is pretty convenient for direct export!

Displaying

We did some research on various solutions to display the data. The first choice we had to make was :

query some service that would build the graphs for us (like Google Graph)
or build the graphs by ourselves.

The first solution initially looked sexier, but then we couldn’t find the flexibility we wanted, and since we export results as Json, we thought* that anyone could actually plug the data we generate for them into the 3rd party* app they wanted.

The second decision was for the “type” of graphs : flash? images? Usually the first one are more interactive, but they require the use of flash :/ Lucklily we found Rapheal.js and its charting library which allows us to generate nice looking interactive graphs by using the browser’s capabilities to deal with SVG and other open formats! We got our winner :) Feel free to look at the source code of the graphs page to see more… Hopefully I can find the time soon to “bundle” that in something nice on github.

Tumblr has a hub

2009-12-12T00:00:00-08:00

There is not a day where the real-time web isn’t growing. It’s now time to change how our web services consume information from each other and we’re quite proud to say that Superfeedr is helping services like Tumblr pave the way.

Today, we’re announcing a hub for Tumblr, one of your favorite blogging platform is publishing their feeds as PubSubHubbub via Superfeedr. I was very impressed when I met David and John (the Betaworks gang is awesome!) : they were both aware and excited about publishing their content in real-time. We’ve been able to setup their hub in a very short timeframe, and it shows how that matters to them.

They’re joining our hub family with Posterous, Backtype, Wordpress (unofficial), Identica (unofficial) and a few other ones on which we’re working. If you want one of your app : get in touch!

Feedburner stats for real-time web

2009-12-09T00:00:00-08:00

It’s been a feature request since day one (almost!), and we’re rolling it out today : analytics. Subscribers wanted to know how many feeds they susbcribed to with Superfeedr, while publishers wanted to know how many feeds from them are used in real-time, as well as how many subscriptions they generated.

It is very important for publishers to know how many apps are subscribed to feeds they produce. When you’re a big media company, it matters a lot to know what content is being consumed and by who. That is exactly what these analytics allow them to do.

Of course, ~~as with any decent analytics solution~~ you can export the data from these analytics, and you’ll soon (when we have the data!) be able to have a weekly view, a monthly view and a yearly view.

Once more, any feedback is appreciated, superfeedr is your app: help us make it better!

LeWeb's Official Bloggers in RealTime

2009-12-08T00:00:00-08:00

When we’ve been selected to be part of LeWeb’s competition this year, we thought it was a great opportunity for us to demo Superfeedr with a simple use case : how can we agregate all the content published by the Official Bloggers in real-time on the main LeWeb’s site?

How-to

LeWeb’s site uses Drupal. The idea was to automatically push notifications from Superfeedr to Drupal so that whenever one of the official bloggers published an article, it is immediately linked (with a small excerpt) on the main LeWeb site.

Drupal doesn’t seem to have a PubSubHubbub subscriber library (yet?¹). However it has an XML-RPC interface. The goal was then to translate PubSubHubbub notifications into XML-RPC. For that I created a very simple Google App Engine application. It basically handles notifications and turn them into XML-RPC calls. 20 lines of code.

To make it a little nicer, I extracted a few information from the standard entry schema : author name and link, as well as categories. With the categories, I did some filtering to make sure we only publish stories that either mention LeWeb or explictly declare that as a tag.

Result

In less than 2 hours (litteraly), I was able to create a pretty good republishing platform:

No polling at all by drupal
Guarantee to have real-time content
Standard schema that allows content filtering and extraction of specific fields

LeWeb starts tomorow and I can’t wait to see you there! Stop by our booth : we have stickers and a few new features to show you!

¹ If you’re a Drupal/PHP developer, get in touch : I’m sure such a module is much needed and should replace the aging “feeds” module.

Results of the contest

2009-12-07T00:00:00-08:00

A little over 3 weeks ago we announced a contest to win a free ticket for LeWeb. The goal was to create an application that uses our Rivers.

A few apps were submitted and I want to thank all participants for their efforts. Several hackers spent days to work on that and we’re quite happy with the outcome : several platforms, multiple languages, different technologies.

And the winners are (in reverse order) :

#3 Superfeedrfall

Superfeedrfall is a simple web interface, built with a long polling webserver written in Erlang called Hydrometeor. The interface shows the messages as they come as well as their source and publication date! Pretty fun and fascinating to watch :)

#1 (ex-aequo) Rss.im’s stream

RSS.im’s stream is another app with the same purpose, but they added a few little features, like the favicon loading. This time, it’s built with gevent and django. It’s open-source, so if you want to learn about the real-time web technologies, you should definitely check it out!

#1 (ex-aequo) AppNotification’s iPhone app

This is a version where you can receive the rivers notifications directly on your iPhone! They use the infrastructure they built for AppNotifications and just added a Ruby listener to the stream that converts the stream into notifications they can deal with.

Luckily for us, Denis, from RSS.im is in Russia, which means that he can’t really fly to Paris. As a consequence, I’m quite happy to grant Fabien from AppNotifications with the free ticket to come with us at LeWeb!

Again, I want to thank everyone who participated, I believe I sent an email or chatted to each of you. You did a great job : THANK YOU! Pleas stop by the Superfeedr booth and come see me for my demo at 4PM, in the demo room. Wish me luck!

The Real-time Web and it's future

2009-12-04T00:00:00-08:00

Marshall have been kind enough, when he wrote the ReadWriteWeb report on the Real-time to ask for my opinion. I want to complete my thoughts as well as give a few hint on this report.
First : go buy the report. If you’re vaguely interested in the web-economy you should have done that already.

On the report itself

Newspapers are dying because of the internet. They’re not fast enough to write about things. Even though blog are less affected, they have exactly the same issue. The fight between ReadWriteWeb, Techcrunch, GigaOM or Mashable to get exclusivity is just a proof that they can’t spend a lot of time writing articles and they also need to write a lot.
There clearly seem to be a premium for the most frequent publisher (less deep though :/)

On top of that, people like me (and probably you, too) have a lot of trouble catching up with all that. That is why reports like this really matter. They bring back “investigation” journalism to blog and that’s great.

Real-time : the peer-to-peer architecture.

For once, I agree with Dave : the Real-Time web happens without waiting. However, I’d complete that by saying that without waiting applies to anything : people (us), but also (and more importantly) : machines.

We used to have clients talking to servers, whether they were browsers talking to websites or web applications talking to other applications via their APIs.
I think the real-time web is taking us to a world where we have a more “peer-to-peer” approach : everybody is a client and a server to everybody else.

In a way, when you blog, you’re a server, and your readers are your client. When you tweet, you’re both a server and client.
When you poll RSS feeds, you’re a client to the website that hosts these feeds, when you use Superfeedr, you become a server because you respond to our notifications (and we’re a client because we send out these requests).
When you use Wikipedia, you’re a client to whoever wrote the article you’re reading. When you use Aardvark, you’re both a client and a server (because you ask and answer equally).

I’m quite happy with this quote from John Borthwick, one of our investors, at Betaworks :

We believe things are becoming more connected. In the future, everything will consume APIs and publish APIs.

In a nutshell, this report is probably what’s best today to catch-up with all the real-time web. It also contains a great list of people you should follow (not only on Twitter) and who are making the real-time web happen.

State of Real-Time feeds

2009-11-27T00:00:00-08:00

As promised last month, here is a small update on the real-time feeds market.

First, I’d like to insist on the method we use to collect these statistics. For each feed, we always try to determine what is the best way to get its content and push it to our users in the timeliest manner.
If you were building a similar infrastructure to ours, assuming you’d support PubSubHubbub, RSSCloud and SUP ¹, here is the breakdown that you would get.

Compared to last month, we have three times more feeds and the proportions evolved significantly.

As you can see, it’s pretty obvious that there are less and less feeds that you’ll be actively polling (still about 75% though).

This is mainly due to the fact that we added white-label hubs to our service offering.

For example, since we have our unofficial hub for wordpress, we subscribe to a lot of RSSCloud feeds on behalf of our users.

Similarly, we now host official the Posterous hub and Backtype hub (among others), which means that if you poll their feeds, you’ll use Superfeedr :) As a matter of fact, we host the hub for 48% of the hubbub-enabled feeds that our users subscribed to.

¹ We implemented many other ways of getting feed updates, including some specific APIs and other techniques.

Thanks

2009-11-26T00:00:00-08:00

Today is the probably the most symbolic holiday here in the US. It’s about saying : Thank You!

Thank you Stephan, thank you to each and everyone of our users, thank you to our advisors and investors, and thank you to all my friends and family who supported me and my unbearable schedule over the past months.

Thanks, and happy holiday to you and everyone in your heart.

Added Title to Feed Info

2009-11-24T00:00:00-08:00

We just pushed a small code update (it should be fully deployed within minutes) to our schema. Of course, this is fully downward compatible.

This has been one of the earliest feature requests that we had : feed titles in the notifications.

In many apps it’s crucial to show the user with the feed title. Up until now you needed to poll the feeds once in while to get that information.
From now on, you’ll get the title in your notifications :)

With PubSubHubbub, it’s part of the element. With our XMPP API, it’s included in the element. Check our schema for a refresher :)

Seed Round

2009-11-17T00:00:00-08:00

That’s true. We’ve finalized our seed investment round. It took us a little more time than expected, but we’re very happy to join Betawork’s and Mark Cuban’s portfolios. Both of them have shown a genuine interest and vision in what appears to be the real-time web. We’re also pretty excited to announce that both Andrew Anker and David Recordon have joined our advisory board.

If you’re following, even remotely, the world of start-up or the one of basketball, you can’t ignore Mark Cuban’s name. Mark founded Broadcast.com a few years ago and sold it to Yahoo!. I am thrilled to have him on board. There aren’t many experienced entrepreneurs our there who have invested in start-ups like Mark did. He knows how to grow a business and we will learn a lot from him.

Betaworks is probably the most active investor in the real-time web. Their portfolio includes names like Twitter, Tumblr, Bit.ly, Tweetdeck, Twitterfeed as well as many others. They’ve proven quite successful at building a network of loosely joined pieces where all members interacts together. I am very excited to be on stage later today at Betaday to chat with Iain and Tony about the death of the web page.

We’re very happy and proud about this seed round, and I want to thank personally Mark, John, Andy, Tony, Andrew and David for their trust.

And now : back to work! (not that we didn’t work a lot lately :D)

Win a ticket for LeWeb!

2009-11-16T00:00:00-08:00

We’re very excited to be part of LeWeb’s startup competition. We will have a few minutes only to demo in front of the jury how cool superfeedr is. Needless to say that it’s pretty hard for an API-based service, and we want to show-case cool stuff created with Superfeedr!

As any other candidate, we got 2 tickets. One for me, and one for… YOU! Well, not all of you, but just the one of you who creates a coolest app out of a new small feature we’re launching today : our rivers.

The rivers will be freely accessible streams of data out of Superfeedr. The first one (and the one that’s in competition), is a sample of our global stream, with at most 1 message every 3 seconds : http://rivers.superfeedr.com:8001

We built this river with the newly released Compp : a small HTTP server that allows you to do Comet Streams out of XMPP traffic. It’s an AtomStream : a never ending HTTP connection that contains ATOM entries¹.

There will be several judges for your apps, including Astro, me and a few other people. The rule is : no rule! Make something, fun, useful, profitable, open-source, downloadable, web-based, or any combination of that! You can use any platform, any other API (you’re even encouraged to), any language (heard of Go?).

The only constraints are :

it must use the river that is accessible at http://rivers.superfeedr.com:8001
your app must be visible (dowloadable or web accessible) on December 4th at midnight PST.
you must write a blog post or a doc page explaining what it is, how it works and how to use it!

We will let you know who is the winner on December 5th and hopefully, you’ll join me on the 9th!

Let the best win!

¹ It’s a very first release, make sure your code is robust enough to deal with errors and disconnections.

SUP over PubSubHubbub and XMPP

2009-11-12T00:00:00-08:00

The Simple Update Protocol is one of the very first protocols that was built to help feed polling. It makes it easy to find the most appropriate time to fetch a feed.

Created by the FriendFeed team, the idea is, for a publisher, to keep track of all the feeds that have been updated in a sup file. Then, a subscriber instead of querying the feed over and over again, just have to query this sup file. If a poller is just polling one feed, that’s not better, but if you’re tracking 1000 feeds, instead of polling them all, you just need to poll that specific SUP file.

One of the issues is that it’s not immediate, these SUP feed have a “period” and contain all the feeds that have been updated in that given period. The standard is 60sec though, so it shouldn’t be too “late”! Another issue is that these SUP can contain a LOT of feeds, so if you don’t have an very fast and efficient way to compare all the entries in this file to see if you’re actually tracking a specific feed, it may be quite “complex” to be sure to get all the updates.

Anyway, I have no idea why I’m telling you this, because, as a matter of fact, you don’t need to implement SUP, because Superfeedr does it for you. Give us a SUP enabled feed (from BackType, Disqus, Friendfeed, 12seconds.tv… etc), and we’ll fetch the right SUP file and you’ll get your content in a “close to real-time” fashion, via PubSubHubbub or XMPP, exactly like we did for RSSCloud!

The Non-Decision makers

2009-11-11T00:00:00-08:00

Superfeedr is a very exciting venture. Yet, I am incredibly frustrated when I am waiting for someone else to make a decision, answer an email or write some code. You know who they are : these partners, customers, providers, lawyers, venture capitalists, etc… all these people you deal with and who take forever to make a decision. I claim they are the worst ennemies for entrepreneurs.

They don’t want to make decisions

And they’re right. They’re not the ones taking risks. VCs don’t want to be wrong, so they’d rather say maybe, later, not sure yet, I need to chat with my partners, when you have revenues… than anything that would look like a commitment. Customers also don’t want to make a decision, because they don’t want to show you that you have a bargaining power, so they will keep you waiting. Even journalists don’t want to make a decision : they want to talk about what everybody else will talk. The problem is : they sometime have a hard time predicting that, so they’d rather wait for a few hints before going out with your story.

Why no is better than maybe

In France, there is a saying that states that “as long as there is life, there is hope”. I’d claim that more than hope, I want certitudes. Entreprendre is all about making an assumption and test-proving it, making another one and test-proving it, making a third one and test-proving it. I say “test-proof” and not “proof”, because in the end, it doesn’t matter whether you’re right or wrong. What matters is that, after the process, you know and you have less and less assumptions to make.

When somebody says “maybe”, “later”, “we’ll do it”… you don’t learn anything, because that doesn’t prove anything. When they say “No”, you know something.

A few tips

As an entrepreneur, you should always make decisions. The Dalaï Lama said : “When you lose, don’t loose the lesson”. The only thing an entrepreneur can lose, is actually the lesson. Make as many decisions as you want, and keep proving them right or wrong. If you wonder if A or B is better, don’t think too much : try, you’ll know.

You can spot non-decision makers pretty easily : they don’t answer your emails (or at least, not right away), they never commit to anything (even the smallest things), they keep using the future when they speak/write. One you spotted one, avoid him at any cost. It doesn’t matter how hard you will try, they will not help you. I wrote code, prepared scripts, documentation, how-tos… etc. If somebody isn’t making the decision to use your app, all this is useless.

On the other hand, every-time you meet anyone who is making even the slightest decisions about your product (like tweeting about it, registering, sending you an email… anything), you have to be very very responsive. Not only they will be great advocates, but since they make decisions, they will help you learn much faster and understand things that would have taken years for you to grasp. You almost have to scare them with your dedication!

Posterous supports PubSubHubbub

2009-11-09T00:00:00-08:00

We’re very excited to say that Posterous now supports PubSubHubbub and became real-time! If you’ve got a posterous blog (and you should!), your posts will now show up in real-time in any feed reader that supports PubSubHubbub on the subscriber side.

What’s even better (for us), is that the Posterous chose to use our publisher services for that, and their hub is hosted at Superfeedr at the following address : http://posterous.superfeedr.com/

We’re working hard to have more and more publishers support this awesome protocol and it’s great that the Posterous team (which I met recently) is paving the way.

If you’re a Superfeedr subscriber user, there is nothing you need to do, you’ll receive Posterous feeds in real-time too :)

Sunday Package

2009-11-08T00:00:00-08:00

A few little news on this glorious sunday.

Async Subscription supported with PubSubHubbub

Several of our users wanted that, we did it. Up until now, we only supported the sync verification for the subscriber’s intent in PubSubHubbub. That was an issue for several people who were debugging with a server that could only accept a single HTTP request at a time, or with hosts like Heroku, which allows a limited (starting at 1) number of simultaneous HTTP requests. You can now use async for the hub.verify param. Please note however, that the dashboard currently uses the sync mode.

A Facebook fan page

Keeping in touch with our friends is very important to us. There is a pretty strong chance that you’re already on Facebook. Become a Superfeedr fan, we’ll give you exclusive content very soon :)

Another Ruby wrapper

The topic of the last XMPP meetup was hacking. I investigated Google App Engine’s XMPP support (and used that to build notifixlite), and I know some friends worked on some other cool stuff. Our friend Jeff, who created Blather, a pretty awesome XMPP library worked on another Ruby wrapper for the Superfeedr XMPP API. Check it out, it’s awesomely simple!

Notifixlite

2009-11-07T00:00:00-08:00

Superfeedr comes from Notifixious. We followed Dave’s advice and removed many features, so much that we found a market! :) However, I am still using Notifixious on a daily basis and I know many other people do. Yet, I am not fully satisfied with it and there are many things that deserve to be changed and improved. One of it’s biggest “disadvantages” is its code-base : it lacks documentation, tests and I am scared to even change a tiny thing because I’m afraid it may break a lot of stuff.

When Google announced that AppEngine was supporting XMPP, I felt it was a great occasion to see if I could give it a shot and I created Notifixlite. It’s a simple XMPP bot that you can use to subscribe to your favorite blogs and receive entries via Jabber/Gtalk. It’s powered by Superfeedr and uses the PubSubHubbub API. Why not the XMPP? because Google’s implementation of XMPP is very very light and doesn’t support IQs. As a matter of fact the implementation is only geared to build bots that can interact with “humans”.

Anyway, I learnt a little bit about Google AppEngine and I kind of felt in love with that, it’s a wonderful feeling to just “write code”, the rest (configuartion, deployement, datastores… etc) being dealt with by Google. I can’t recommend more to look into it for anyone with a project that can fit there. I also ~~learnt~~ played with Python, and I am sure my code needs a lot of improvements!

Please fork Notifixlite, and, as would the Beatles say : make it better!

Funny Feeds

2009-11-06T00:00:00-08:00

Our business cards say : “We make something stupid so that nobody else has to do it”. And it’s true : we do a lot of things so that nobody else has to. While debugging some code last weekend, I checked at some feeds that we had trouble dealing with. If you use Superfeedr, you won’t have to deal with that, and that’s good, because these things can break your feed parsing module.

Never Ending HTTP responses

This was actually one of the very first things our parsers encountered. Want to see an example : check out the six-apart Atom stream. Technically, this is perfectly legit, except that if our parser starts to get content from there, well, it will eat more and more content, and will eventually die of indigestion. When we had only a handful of servers, we had the problem with some hacker implementing a similar thing on his servers (with no content, just an un-endind HTTP connection). This ended-up keeping our parsers busy to such an extent that they wouldn’t parse anything else.

It was an easy fix : timeout on active HTTP connections.

6847 entries feed.

You read that correctly. That is 6847 entries, in one feed (that’s fluctuating though), and it weights about 2MB. As such that’s not really a problem. We fetch this feed in more or less 5 seconds. However, to identify new entries, we need to compare each of them with a “cached” version. Comparing 5, 10 or even 100 entries is not a big deal. Comparing several thousands can actually take up to a few seconds (we store these entries into Memcache). Eventually, this feed was taking forever (20, 30 seconds or even a few minutes) to be fetched and parsed. We built our parsers such as if a feed takes too long to be processed, it’s dropped. This feed was eventually never fully processed.

We increased the allowance on the feed processing. That was not enough : many feeds could not be fetched anyway, we were just spending more time on them. We finally decided to limit the number of legit new entries in a given feed to 500.

Permanent redirections

That is a much harder one. The HTTP protocol has stuff like redirections. A redirection is just a way for a server to say “what was here is now there”. There are 2 types of redirections : Temporary and Permanent. When it’s temporary, we don’t even keep track of it : our parser follows the redirection and gets the content where it is. When it’s permanent, it’s different. We follow the redirection, but we also save the new feed url and assign it to the subscribers of the initial feed. We still keep the ‘old’ feed, just in case the permanent is not really permanent (and because we hate deleting stuff), but we reduce the polling frequency to a minimum. Some feeds, however make our life hard in this case. This feed from VMWare is an example. If you open it, you’ll see that it permanently redirects to a url like this one and if you load it again, you’ll see that it permanently redirects to another feed… and so on. In other word, this feed permanently redirects to different urls.

We haven’t found a good solution for that yet. We hate to do it, but we basically clean these redirections once in a while to avoid having thousands of them.

As you can see, building a polling infrastructure is full of traps. There is not a day where we don’t find new challenges. It is very interesting and we like that. However, there is absolutely no valid reason for another person to spend time and money solving the same problem.

That doesn’t mean either that the solution we come up with are the best; and that is why we’re sharing them here! Your ideas are our food…

Information Superhighway

2009-11-05T00:00:00-08:00

Because we provide a service to developers and businesses it is quite difficult to “show off” to people without prior knowledge of the technology. Julien built our website with quite some interactivity, but it is still rather hard to understand because you’re going to see the gory protocol in detail.

The well-standardized XMPP with its many implementations has always made it easy for developers to use us. There’s a vast amount of tools and libraries out there. That makes it easy to put together a demo consumer of Superfeedr notifications in very short time.

So we want a graphical app but not restrict ourselves to one platform. The web browser, where you don’t even need to install anything, just click and everything’s in a sandbox is the solution. That’s why it has to be a webapp.

How it’s built

The nature of our service is very dynamic and content must be delivered quickly. That requires JavaScript. Despite not being affine to that language, I’ve always been intrigued how AJAX, nowadays a common web technology, can be used with a tiny glue of BOSH to run an XMPP client in just your browser.

I took StropheJS, easened XML and DOM handling a bit with JQuery, and made a really tiny webapp: Notification Superhighway.

Log in, subscribe to feeds (or even Twitter searches), and get amazed by that millions of content producers on the Information Superhighway.

Running it yourself

Pull the code :
git clone git://github.com/astro/superhighway.git

cd superhighway

Get the StropheJs code:
git submodule init
git submodule update

Proxy the http://localhost/http-bind/ to use http://superfeedr.com/http-bind/. The browser’s Same Origin Policy will allow script access only to the local domain for good reason. To proxy the BOSH entry point I included a configuration file for nginx. ¹

./start.sh (you may need to use sudo) ²

http://localhost:8080 should do it :). The Interface clearly needs a little work! Please fork Notification SuperHighway :)

¹ On a Mac, I suggest installing Nginx with sudo port install nginx.

² On a Mac, you will need to comment the use epoll; line of the nginx configuration.

Post Mortem 02/11

2009-11-03T00:00:00-08:00

Superfeedr went down yesterday evening. We immediately saw that it was a database performance issue (MySQL), given that the load on our MySQL host was about 30+. Since we have 2 MySQL hosts (Master – Master replication), we switched to the second one, hoping that was just a problem on the host. No luck.

Finding the issue

Up until yesterday, our entry detection was was done by storing in Memcached and MySQL an unique identifier by entry. If the item was not present in Memcached, it would be inserted into MySQL, which has a unique index on these unique ids. The goal of that is that a check on Memcached is much cheaper than a check on MySQL. We had then a maintenance script that would clean up the entries that are old enough to not be considered new again.

When we investigated a little further yesterday, we found that the it was actually these INSERT queries on MySQL that were taking much longer than they should, and, worse, much more often than they should.

Logically, we checked our Memcached and found out that our caching mechanism didn’t work as expected: whether the unique entry was present in the cache or not, we would send the entry to be saved in MySQL. We fixed that. But the problem was still the same : way too many INSERTs in MySQL.

After checking Memcached into more details, we figured out that our keys were “disappearing”. Looking at Memcached’s FAQ was helpful. We indeed had a flush somewhere, that we had forgotten about. Removing the flush was good. The load on MySQL started to decrease, but not enough.

There was a problem with MySQL. Another proof of that is that the 2 issues we had in our cache system had been here for weeks.

After checking the entries table, it had about 65 Million rows, 12GB Data and a 15GB Index. Every time we insert a row, it has to check the keys. And since we’re inserting about 100 new rows per seconds, even small operations start to become too heavy, specially if they involve disk access (we don’t have 15GB RAM on the MySQL host).

We had to clean this table. It was easy to see that some feeds had more than 1M entries in MySQL. We started some queries to clean them up. It took 7hours and didn’t complete. We needed to be back up faster than that.

Fixing the problem

Here is what we did : we changed the way we store entries in MySQL. Instead of cleaning them only once they’re too old, we only keep in the cache a handful of entries : twice the size of the feed actually, so that we don’t store millions of useless entries, and we clean these entries each time we add some.

The “reboot” process comes next :

We then truncated the entries table,
We stopped the notifications,
We restarted Memcached to clean any cached entries,
We forced all the feeds to be ran through our parser once to populate the entries table again.
When that was done, we restarted the notifications, and Superfeedr was back up.

There is a small risk for you to see duplicate entries (if the feed couldn’t be parsed/fetched during the populating phase), but that should stay minimal.

What did we lean?

MySQL is not the right tool for what we do. Unfortunately, we haven’t found any tool that really complies by all of our requirements. We studied several of the NoSQL alternative. We’ll do that again in the coming days.

A problem can have several causes. The first one you find might not be the good one.

You have to take drastic steps earlier : trying the “soft” methods may sometimes just be a time loss. We should have not even tried to clean only parts of the entries table and that lost us about 7 hours.

We know we may pay a price for these lessons in terms of customer and conversion rate. We also know how valuable these lessons are and this also validates our biggest assumption that nobody else should build a similar infrastructure. Of course, not everybody would have fallen into those traps, but at least, if you use us, we did that for you, too.

Please, let us know what we can improve and how to be better. We also learn from you.

EventVue's Chatter

2009-10-23T00:00:00-07:00

Meeting the right people at conferences is always very difficult. Even if you’re the most extraverted person in the room, most of the time, you just don’t have enough time. EventVue is providing a very elegant way for that by helping you prepare your conferences in advance.

When we started Superfeedr, Josh helped us a lot by providing a good use-case. We said it in the past, but Superfeedr is yours, and you can thank him for his feedback. We’re now thrilled that EventVue uses Superfeedr for its “chatter” functionality.

The chatter is a stream that contains any message that was posted by people at a given conference : they write on their blogs, they tweet, they post to Youtube, Flickr… it all ends up there. It’s like Twitter Lists on steroids.

EventVue uses PHP and the PubSubHubbub API. I’ll just let Josh saying what he likes about Superfeedr :

We’re really grateful to Superfeedr for the service they provide. Polling thousands of feeds was beginning to be a huge distraction from working on our core product. Superfeedr allowed us to stay focused on the features that matter while still being confident that our feeds will arrive in a timely and reliable manner.

What else…

Smacksale

2009-10-21T00:00:00-07:00

There is not a day where people don’t ask us about applications that use Superfeedr. We have been truly amazed by the quality of some of these apps and we thought we should share them with you. Today’s example is Smacksale.

Smacksale is a simple aggregator of the best e-commerce deals out on the web. You can vote up, down or comment these deals : simple, efficient and useful (I just found a great deal for Gogo Inflight ; can’t wait to try it).

Built for the 2009 Rails Rumble, Smacksale was created by Adam, Pradeep, Brendan and Paul. It’s a RubyOnRails application, hosted at Heroku and they use our PubSubHubbub API to receive the new deals from other services. They extract the information they need from these notifications and build their own tag-cloud out of them.

PubSubHubbub for Everybody

2009-10-11T00:00:00-07:00

Today is an exciting day! We’re are at the ReadWriteWeb Real-time summit. If you’re here, come and ping us, I wear a yellow sweatshirt.

It’s also exciting because we’re launching a new version of our website with many many improvements over the previous iteration. I’d like to thank my friend Beau for his great help, as well as David. We’re perfectly aware that we have a long way to go, but we’re moving toward the destination.

If you go check this new iteration, you’ll immediately notice the shiny orange buttons. The subscriber part is nothing new, however, the publisher part is brand new!

We’re announcing today that Superfeedr now offers white-label hubs. What is that? If you read this blog, you know how crucial it is to engage your content consumers with real-time data, but building a hub is not the most easy thing to do. Again, we’ve done that for you, so that you don’t have too.

We’re introducing to options :

A free version, where your app’s hub is accessible at http://appname.superfeedr.com and where you basically just need to add the ‘discovery’ tag to your feeds. Check out our Wordpress and Identica hubs.

A for profit version, where you can use your own domain, where the subscribers will not even know that this service is provided by superfeedr.

We’re very excited about that, and we already started working with several great teams to integrate hubs into their apps. What do you think?

PS: As I write this, I see that our blog’s layout is not as shinny as I’d wanted :/ I am working on that…

State of Real-Time feeds

2009-10-10T00:00:00-07:00

Real-time is the new hotness. At Superfeedr, we’re lucky enough to have many feeds we’re fetching, some of them being PubSubHubbub enabled, some of them being RSSCloud and other being none of that. We’re in a good position to assess how many feeds are using which technology. Being french, I appreciate showing you this ~~pie~~ camembert chart, which shows how we get the data for the feeds in our database.

Please, assume that we have a bias: we’re still pretty young, and our bigger users give a “taste” to the type of feeds that we have.

If you’re building your own feed polling infrastructure, – but you obviously shouldn’t -, you can save:

7.1% of your work by implementing PubSubHubbub subscriber,
0.3% if you’re an RSSCloud subscriber.

Once you’ve built that, you can easily do some math and see how much it costs you. Then, use our hub (it should be trivial since you have implemented PubSubHubbub) for all your feeds, and start saving!

We said a few weeks ago that we believe that the amount of real-time available feeds is pretty much stable. What is RSSCloud or PubSubHubbub now was either XMPP or AtomStream weeks ago.

We’ll see in the future how these ratios evolve.

Real-Time web browsers

2009-10-08T00:00:00-07:00

The web was geographical (document-based) until very recently and it is now evolving faster and faster to add a “time” dimension. Back in the days, Mosaic was the very first tool to actually browse these sites from a site to another. Unfortunately, all these browsers were built to deal with documents, not with a stream. The best proof of that is the presence of a “reload” button!

Twitter and facebook are definitely making real-time web mainstream, but they are just “content” producers, we need the browsers to consume that content. And before the browsers, we need the protocols (I’ll talk about that later!).

There are many tools that start to look like this real-time browser : Seesmic is paving the way, Tweetdeck is another example, but Growl also presents some very interesting features. iPhone applications like AppNotifications can also be considered as real-time web browsers.

Here is the set of functionalities that I’d like this stream-browser to implement :

Stream-agnostic : Mosaic could display any web site… not just and handful of them. Seesmic should be able to receive content from any stream, not just Twitter or Facebook. I want my RSS feeds, I want my emails, I even want Google news… etc.
Context-Aware : I want the stream to follow me on the desktop, on my phone, or even by fax (yeah!). If I am on the laptop, it should show the message there, if I am watching TV, it should show it there, and if it’s 3AM, please, do not notify me.
History : I want to be able to travel back in time.
Re-Stream (share) : exactly like the Re-Tweet functionality on Twitter. The Real-time browser must not be the ‘end’ of the stream : data must flow from there to my friends, to other services that I use… etc.

I know all that sounds quite ambitious, but we’re really not too far from there. All the bricks are ready actually, we just need some glue between them.

RSSCloud and PubSubHubbub

2009-10-06T00:00:00-07:00

“Why make things simple when you can make them complex?”. It was probably just too easy to have a common open protocol that everyone would share and use for push notifications of feed changes. At Superfeedr, we chose PubSubHubbub, for obvious reasons, and we’re quite happy with it.

Yet, some publishers have made a different choice and since it always eventually comes down to the customer’s experience, we had to implement RSSCloud. Superfeedr is now an aggregator. As a user, if you give us a feed that is RSSCloud enabled, then, we will subscribe to the right cloud and you should get updates from this feed in a few seconds as well (exactly like with PubSubHubbub).

The awesome part of that news: since we use PubSubHubbub as our API (we’re one of the few independent hubs), this means that if you use us, you don’t even have to care about RSSCloud. Once again, we do all the “hard” work for you :) Just implement the PubSubHubbub subscriber part, and you’ll get the RSSCloud feeds through that as well.

Time vs. Space : how do we apprehend information?

2009-09-19T00:00:00-07:00

There are several ways information comes to us, but they are either mostly either geographical or temporal.

By geographical, I mean that we basically lay-out information and draw connections between them : a map can help us understand how a city is organized, how a product is conceived, but also a conference speech is usually logically organized with bullet-points, transitions, etc…. Most of the web information is currently available on a geographical basis : a web-site is a collection of pages linked together, and even pages are “flat” : related information are usually displayed together. Comments are show under a main story, images are included in the body, etc… I fully agree to that extend with Chris and Jyri’s article about the web being document-oriented.

We mostly use the this geographical approach because the human brain also uses location to relate things together : our thoughts are linked and that is how we end-up remember things. Of course, we still have some sort of “index” which are the main things from which others are linked, like our name, the place we live in… but all that started to be connections from stuff we perceived and these connection became so obvious (our brain kept taking them), that we forgot how they happened¹. Everybody started with a mnemonic to remember his phone number, but nobody needs it anymore after some time.

The real-time web is all about using time, instead of links. The connections between elements don’t matter so much anymore. Unrelated tweets stack up. Our brain is not so good at stacking things. It’s even pretty bad at that, because in a stack it’s hard to find links between things. One of the most common examples is names. How often have you been introduced to some people in a group, just to figure out 10 seconds later that you don’t even remember anyone’s name?

Do you remember the sequence memory game we used to play when we were kids? You hear a random number between 0 and 10, and must repeat it, then you hear a second number and must repeat both the first and the second… etc. It is usually very hard past 8 to 10 numbers. It’s modern counterpart is just remembering a phone number.²

The fact that our short memory is very bad (or at least worse than chimpanzee) can also be related to that as well : we have trouble dealing with short unrelated messages coming at a high frequency, or at the same time (like numbers on a grid).

As humans, we can deal pretty easily with complex information as long as we can tie things together and lay down this information in a geographical manner. Dealing with a river of news, with a tweet timeline is, on the end, quite stressful, because this information is time-organized.

I am not sure it’s one of the problems we faced with Notifixious (as there are many others), but I know for sure that we will either have to accept to lose parts of this stream of information or train computers to help us relate things in the real-time stream.

¹ I really wish I could remember how I initially linked my name to me; this must have been a huge breakthrough!

² The most common technique to split the number in 2 or 3 and ask people to remember a part each is a way to actually transform a stacked-information into something more logical, where we can draw connection between numbers.

The New Atom Schema is up!

2009-09-18T00:00:00-07:00

We deployed yesterday a new code for our parsers, and it now includes a more complete schema that will allow you to do a lot more things with superfeedr!

Content or summary?

Up until now we were considering that anything was a summary, which wasn’t really accurate since some feeds actually offer the whole content. We fixed that. If the entry contains the full content, you’ll have a <content> tag, if not, then, it will have a <summary> tag.

Links

The biggest addition is the fact that <entry> elements now include all the <links> of the original entry, including attachments. Of course, to differentiate between them you can use the "rel" value, as well as "type" which gives you the mime type. For example, here is what you’d receive for this flick RSS feed :

As you can see, there are 2 <link> elements, one for the entry (the page that contains the image), and one for the image itself. These links are various and may include links to the comments section, to the RSS feed of comments, to any attachment, to related articles… etc.

Authors

Last, but not least, we now support author informations. There can be several authors in one entry. For each author, we provide (when available) a name, an email address and a profile URI.

If you’d like anything else to be added, please let us know in the comments. With the current information, we estimate we cover 80-90% of the original entry content. Also, if you’re not satisfied with our mapping, please do not hesitate to suggest fixes as well.

Real-time RSS notifications on your iPhone

2009-09-11T00:00:00-07:00

I have always been a great fan of the iPhone (I had mine on June 29th 2007 : iPhone D-day), but I haven’t found many applications that were making good use of the “notification” feature that Apple added in its last iPhone OS, until today.

AppNotification is an app that allows you to receive any notification. Its creator, Fabien just spent the last day interfacing his app with Superfeedr and the result is here today : you can get real-time RSS notifications on your iPhone.

It’s quite easy :

First, buy the app from the iTunes store : $1.99.
Then, create an account and enter your credentials into the App.
Add the Superfeedr application
Enter a list of RSS feeds which will be pushed to your iPhone
… wait for notifications to come!¹

I’ve added a few RSS from Spy, as well as the weather in San Francisco or the RSS for Superfeedr’s delicious page, but you could also add any RSS that you’d like.

How cool is that?

¹ If you have a superfeedr account, you can generate notifications to the dummy feed, in the console, as long as you add it to your feeds into AppNotifications.

The little lie about Real-time Data

2009-09-10T00:00:00-07:00

It is very exciting to see all the jazz about real-time data being available. It is however a little disappointing that the RSS vs. Atom war is starting again through the PubSubHubbub vs. RssCloud conflict (read Techcrunch’s post about that). Wordpress said they will adopt RssCloud, while Google massively adopted PubSubHubbub and SixApart showed strong interest for the latter.

The only little lie about that is that almost all of the data that you can access through these hubs or clouds, was actually already available through other Real-time (or close to real-time) means. SixApart has had an AtomStream for years, Google implemented the changes.xml syntax a few years ago and Wordpress announced last year that they had an XMPP stream with all the blog posts and comments.

Again, don’t get me wrong : this is really exciting that from having 1 technology for each provider we now have ~~almost~~ the same protocols across the board, but I think that proves also why we need services like Superfeedr : a huge part of the content that is published daily is not being pushed to anywhere.

Again, pushing your content to reduce the “polling” load makes sense, but we strongly believe that the vast majority of the data out there will probably never be pushed to any hub of cloud, and that’s precisely where Superfeedr is so useful!

What API to choose

2009-09-08T00:00:00-07:00

Superfeedr is technology-agnostic, which means that we don’t force you into any API. We use standards so that our users aren’t locked in any way to our service. However, being agnostic doesn’t mean that a solution is not better than another one in your specific case.

Actually, we even believe that in some cases, it is better to use XMPP and in some it is better to use PubSubHubbub. We creates this small “decision-chart” to help you decide with yes/no questions. Of course, this would apply to 99% of cases and there will always be times that this chart is wrong, but I am pretty confident that it should work for you.

What do you think?

Getting Started with PubSubHubbub

2009-09-02T00:00:00-07:00

To my knowledge, Superfeedr is the only non-google PubSubHubbub out there. Let’s see why and how to use it.

Why using PubSubHubbub?

Superfeedr provides 2 API. Here are some reasons why you’d be better off using the PubSubHubbub API:

You don’t want to run a background job
You have an existing Web application
You already use PubSubHubbub with another hub.

And here are a few reasons why you may want to avoid the PubSubHubbub API.

Your endpoint (where you will receive notifications) is *behind a firewall
You want to receive a notification each time Superfeedr fetches and parses a feed for you
Your endpoint(s) are either desktop applications or embedded applications

What do I need to do?

The PubSubHubbub API will allow you to receive notifications from us when we identify new entries in feeds to which you subscribed. Thus, there are 2 things that you need to do : subscribe and implement a “receiver” to deal with notifications.

Subscription

The subscription process is not obvious, because the hub needs to verify your intent. Here is how it works.
Our hub is located at http://superfeedr.com/hubbub

Send an authenticated (with your superfeedr login/password) POST request to our URL with the following parameters :
– hub.mode = subscribe|unsubscribe
– hub.verify = sync
– hub.callback = http://domain.tld/your/callback
– hub.topic = http//feeds.you.want.to/subscribe
When our server receives this POST request, it will send back a GET request to your callback URL, with the following :
– hub.mode = subscribe|unsubscribe
– hub.topic = http//feeds.you.want.to/subscribe
– hub.challenge = a-random-string
If your server echoes the hub.challenge parameter (along with a 200 response), the subscription will be accepted. It will be refused in any other case.

To debug the process, you can use this hurl to send subscriptions requests (don’t forget to adjust the values of the params, though). Curl is another great option.

Receive Notifications

Your application must “deal” with the notifications. Notifications are POST requests sent to you callback URL, the new entries are in the request body. You then have to use an XML parser to fetch that body. Since we use standard ATOM, any parser that can deal with ATOM will be great.

If you’re using a web application framework like Rails, Django, Merb or Symfony, it should be quite easy to register a “route” connected to the right Controller, in charge of parsing this body.

Existing libraries and plugins

The great advantage about using a standard protocol, is that you can re-use any “code” that’s been written for other hubs. Here is a small list of existing libraries that can help you deal with the subscriptions and handling notification from your web app. That should all be fairly easy to implement.

You can also find several PubSubHubbub repositories on github, some of them claim to implement the subscriber part of the protocol.

Finally, please take the time to read our PubSubHubbub documentation as this blog post is just a “getting started” guide.

Real-time search

2009-08-29T00:00:00-07:00

Yes, I am pissed off I am not at Foo Camp, but I am working to be there next year, bear with me. I really wish I could be there because I think we’re making the term “real-time web” as the new “web 2.0” : a brand as Tom calls it. “Blahblah is doing real-time”, “Blihblih has a real-time approach”… And this is bad : when words lose their meaning, concepts get over-looked. This is not just playing on words.

Real-time is not instantaneity. Real-time means time-constrained. That’s it. That constraint can of course be low (up to the second if you want, even).

Based on that, a real-time search engine would be a search engine that would be able to guarantee that the results of your query happened within an X seconds/minutes/hours/days.

If I search for ‘Foo’ on twitter, Twitter tells me that the constraint is ‘now’ (See what’s happening — right now.), I am guaranteed that the results have been posted ‘now’ : I’ll leave the now definition to Twitter :D.

By studying a query over different time-constraints (within 1 year, 1 month, 1 day, 1 hour), we can probably get very interesting results : maybe something like the log of Google’s Database of Intentions. Applied to Twitter, that would allow us to learn a lot on how we communicate, react to “information” and propagate it, or even correlation between things people are talking about.

Remember when Google announced they could predict the “flu trend” better than the CDC? Well, my bet is that Twitter can do that even better than Google, and for maybe less “common” things. A real-time search on heading into Twitter, coupled with a hint of geo-localisation and a pinch of semantic analysis would probably be quite powerful to know (and predict) what the traffic looked like at a given place, on a given time.

Want to predict the “economy”? Track down keywords like “laid off/let go”, “hired”, “profit”, “loss”, “chapter 11”…

A new Architecture

2009-08-25T00:00:00-07:00

No post for almost 2 weeks : we’ve been incredibly busy. We had a lot of instability in the past 10 days, in conjonction (but not related) to significant API changes. We’d like to apologize for the few times when you had problems accessing our service. We’ve done our best to limit these downtime periods and most of the time they only affected subscription/unsubscriptions : notifications were still going on, as usual.

In that period, the number of feeds in our system increased by 50%, most of them through the PubSubHubbub API. This significant growth in the HTTP traffic made light on the weaknesses of our HTTP architecture. The HTTP API had a direct access to the MySQL database. This was quite bad design from us, as both the XMPP and HTTP were accessing this database and basically doing the same thing : subscribing and unsubscribing users to feeds.

As I am a true DRY lover, I just hated the fact that we had to create and maintain 2 components doing exactly the same thing and I’ve been looking for ways to change that.

Here is how our architecture looked like until this morning:

The XMPP App would deal with all subscriptions and unsubscriptions, as well as notifications from users using the XMPP API. Nothing changed there.

The Web app would do the same with both the website (you can add/remove feeds through our website) and the PubSubHubbub API. It was a plain vanilla Rails application.

It was simple to understand, but not so clean and probably not very easy to maintain, since we have so much duplicate information. I was also “scared” by the scaling issues we might have had to face on both sides, but I knew that our XMPP app can scale pretty easily as XMPP components are load balanced by default by EJabberd, which means that we can scale horizontally in a mater of minutes (Chef is great!).

Here is how it looks like now:

At first sight it looks more complex, and you’re probably right, but it’s also more coherent and much much much dryer. What changed? Basically, both the web app and the PubSubHubbub API became XMPP clients, as they now both access the XMPP App to add/remove/list subscriptions. In both cases, this is BOSH magic.

Bosh is and HTTP extension that allows bi-directional streams – very much like comet -. XMPP has a BOSH extension, that has been implemented in eJabberd. Once you’ve got this stream, it’s pretty easy to transmit XMPP traffic over HTTP. A key feature is that these connection are “stored” on the HTTP side of the XMPP server. It means that as long as your client remembers the session id of this stream, you can re-connect to the XMPP server and take the connection where it was left.

The web app : Strophe

Strophe is a very well done Javascript library for XMPP. My friend Jack wrote it and it’s awesome. Among its key advantages : a simple syntax, very close to Jquery, and an “attach” feature that allows the application server (Rails in our case), to start the BOSH stream and then hand it over to Javascript so that user credentials are never sent back to the client. That is exactly what we used for the web application. You can now list, unsubscribe, and subscribe to feeds through a simple interface that uses our XMPP API. How cool is that?

PubSubHubbub : Proxying

PubSubHubbub is an HTTP API. That was a little trickier, because obviously, running javascript on the server side doesn’t make much sense. However, like any HTTP API, we can of course “proxy” it from Rails. That is what we did.
Here is what happens when you subscribe or unsubscribe with the PubSubHubbub API :

on the server, it starts an XMPP Bosh session with your credentials
it sends the corresponding XMPP query,
it translate the XMPP response
it returns it to you as an HTTP response.

Of course, there is a performance cost to that, but it’s not that big, as our HTTP server is "geographically"" very close to the XMPP app. Also, at the first PubSubHubbub query that you make, we store the connection into a Memcached server, which allows faster processing on the following requests. In terms of results, we can server any PubSubHubbub query in something like 200ms to 500ms. It is slow, but don’t forget that we also check acceptance of the subscription/unsusbcription (by sending an HTTP GET request) on your side, and that accounts for the biggest part of that time, actually.

We wrote it earlier, but we’re true believers in the fact that web apps are about to see a massive shift in their architecture, and we think that XMPP based apps, where the browser is just an XMPP client are going to be a significant part of that trend. We hope Superfeedr will be an example that parents will show their kids, when they teach them about web apps ;) (Other cool examples include Collecta or Present.ly …)

API changes

2009-08-08T00:00:00-07:00

With more and more users, we get more and more feedback and we’re slowly but surely improving our service to take into account all that feedback. As a consequence, every once in a while we need to push a new codebase with slight changes in the API. Hopefully these changes should be backward compatible, which means they shouldn’t affect any of the existing applications.

This changes will be released on, or after August 15.

Subscription API changes

Up until now, subscription end-points where “user-centric” which means that you could only have one JID or one web-hook for all the user’s subscriptions. From now on, you will be able to have a specific endpoint for each subscription :

A different URL for each of your PubSubHubbub subscriptions. Technically, this means that Superfeedr will implement the full specification, including the subscription verification. That requires a little bit more work on the subscriber side, but it also gives you more flexibility with different URL for different “types” of feeds, or even a simple load-balancing on the subscriber side.

A different JID for each of your XMPP subscriptions, as long as the subscription JID matches the IQ jid. If the IQ jid (in the “from” attribute) is:
- a regular JID, like user@server.tld, you will only be able to subscribe with user@server.tld
- a “component jid” like component.server.tld, you will be able to subscribe with any-user@component.server.tld.

As a consequence you’ll be able to have both PubSubHubbub and XMPP/PubSub notifications, even for the same feeds. But, don’t forget : we charge by the notification : 2 subscriptions to the same feed with different endpoints means you will be charged twice for the 2 notifications.

By default, for all the subscriptions prior to this change, we will use by default your user-defined endpoint.

Subscriptions listing changes

This is only valid for the XMPP API since the PubSubHubbub doesn’t define any subscription listing API.
We will send status information along with the feeds, like in this example :

Similarly, you will receive the feed information upon subscription with the XMPP API.

Please, note that, contrary to notifications, the <status> will not include a feed attribute, as the feed url is already included in the <subscription> node attribute.

We appreciate your feedback and are committed to implement as much as we can in a non-destructive way to avoid any compatibility issue. Please, if you see problem in any of your apps, do not hesitate to get in touch with us and we will shortly find a solution.

Performance monitoring with collectd

2009-08-07T00:00:00-07:00

One challenge we face when working in the Cloud is knowing when to scale up. That means we need some obvious figures telling us we need more resources.

This requires performance monitoring. From the many tools around we chose collectd. This daemon is built on a very sleek plugin infrastructure, allowing it to run with only the functionality you actually need. If you want to know how clean C looks the collectd source code is worth a look.

Contrary to a lot of other monitoring software, the watched hosts don’t have listening network sockets but only send UDP packets to a central server. It’s not necessary to control any data gathering from the outside, thus it avoids potential remote security holes.

The primary use of collectd is monitoring of operating system resources and this is done very efficiently by its native plugins. There’s no expensive fork() for calling other programs. If you still need that, there’s a exec plugin for that.

Once you figured out which host is responsible for gathering all the data, you can configure collectd’s network plugin to open a listening socket for receiving other hosts’ statistics.

To make use of the collected data you’ve got the rrdtool plugin which writes locally-generated or remotely-received values into the round-robin database files of the popular and proven rrdtool. Pay attention to the configuration of that plugin: once you start monitoring dozens of hosts with hundreds of plugins, rrdtool will randomly write to the many RRD files. There are some tunable parameters and the rrdcached solution.

You can generate your pretty graphs with the rrdgraph utility. This is being done by the included collection3 CGI scripts. Unfortunately, this web interface is rather Web 1.0 and allows basic browsing of your graphs only. We hope for the advent of some alternative projects to allow more convenient viewing of statistics.

Because we do not only want to watch host resources but also the quality of our processes we have developed the ruby-collectd library. Basically, you just define a collectd server on application startup and sprinkle your code with statements like Stats.counter(:job_done).count! 1

ruby-collectd does not require collectd, so you don’t need to install it locally just for watching your own application.

Harper's friends

2009-08-05T00:00:00-07:00

There is already a lot of fun stuff that people are doing with Superfeedr and we thought it would be interesting to share them here. Of course, if you have any story, you can either write it on your blog and we’ll link to it (on twitter or here), or you can send it by email and we will be more than happy to post it on this blog .

Harper was among our very first users, and even though he told be he was interested by XMPP, it seems that he fell in love with PubSubHubbub. Over the course of last week end he created a small app called Harper’s friends (Oh Yeah!) which basically aggregates the content of his friends twits, pictures and blog posts on a page. You can also show items by users or by type, like a standalone friendfeed if you want.

All the polling content is done via Superfeedr and he uses the PubSubHubbub API to notify an AppEngine app. I guess the fact that you can’t run XMPP apps on appengine (yet) was one of the reasons he chose the web based API.

One of the key reasons to use Superfeedr is that you don’t have to deal with (or even care about) building a polling infrastructure. Believe it or not, but I can bet Harper would have spent easily 2 or 3 times the time he spent if he had to implement that, and he would probably not have had the same kind of results that we have (probably because that would be overkill).

Focus on what you do best and we will focus on what we do best. This way, the world will be awesome.

Scale with Chef

2009-08-02T00:00:00-07:00

This will be a long post, but it deserves some time. Go get a glass of water, breathe deeply and let’s dive!

The why?

Even though we use “smart” techniques to get content from the feeds in real-time (or close), we still need to do some polling (remember, we’re doing something stupid, so that you don’t have to). To achieve that, we have a distributed architecture where workers can ask dispatchers for feeds to parse.

From there it is somehow easy to scale : more feeds to fetch? add more workers. We’re building a system that, at any given time knows how many workers are (and will be) needed. Once we know that, we just have to fire off some new workers (or kill unneeded ones), and we need this to happen in something like 5 minutes.

Theory

You should really check out this ~~long~~ presentation from Ezra and Adam

They say that scaling usually has 3 steps :

bootstrapping : it is the act of ‘acquiring’ resources. Basically starting servers at a given IP, with a base (naked OS), and hopefully a way to connect (ssh).
configuration : the goal here is to change a “vanilla” server into something operational.
command and control : once the server is configured and running, you still need to send him specific instructions and have feedback on how it performs.

There are several options for the configuration. The most basic one is to do everything by hand (and try to repeat for as many servers you need to deploy). Another one is to deal with an “image” (a ghost if you want), but then it’s quite hard to improve it and evolve it. At superfeedr, the option we chose is to use Chef.

Chef

Chef allows you to define a set of specifications (packages to be installed, files to de deployed, options to be configured…) of what our server should like when it’s ready. There are a few advantages to that : it’s stateless, it is idempotent, it is repeatable… etc.

There are a lot of ways to use Chef. Here is ours. It’s probably not the best for everyone (and maybe not even for us), but playing with it gave us some experience and we’re sharing it with you.

Chef is a client-server architecture, where the client is the node to be configured, which implies several things. The first one (and most impacting) is that everything happens from the client (called node). If you used to do your configurations by hand (or using scripts), you were not doing it from the client. Another consequence is that you need to have your clients to be ‘chef-ready’. This might (and will) sound weird, but you need to have some basic configuration on your servers before you can start to do the actual configuration. Luckily this configuration is rather basic and can be automated by using a script like this one.

Cookbooks

On the chef server, you would define a set of cookbooks. Cookbooks are collections of related recipes. For example, a cookbook can deal with installing the “build-essentials” packages, the apache server or git… etc. For each cookbook, you would have a set of recipes. For example, the git cookbook, may have recipes such as : install client and install server.

Technically the cookbooks can be used by different users and/or for different machines in your architecture. Actually, you want them to be as generic as they can.

Roles

Once you have your cookbooks, you should define roles. A role is a “type” of node you’d like to configure. Technically, roles are just sets of cookbooks. A role for a blog application would include recipes for Apache, MySQL, and Wordpress for example. A Rails application may have an “MySQL server” role, a “Memcached server” role, and a Web application role.

Attributes

To differentiate a cookbook for 2 nodes, Chef has attributes. An attribute would be for example the name of your server, or the port used by apache, the names of the gems you want to install, the user who runs your apache server… etc. Of course, you can define attributes at several levels : inside a cookbook, they would be the default values, or at the “role” level and even, at the node level. For example, a node IP is clearly at the node level, but the login/password to an external MySQL database can be set at the role level, while the apache cookbook should probably set the default http port to 80.

Got it? Of course, this clearly a “crash-course”, you can find a lot more information in the Opscode wiki, as well as in the #chef room at irc.freenode.net. Ezra also wrote a great blog post that you should read as well.

Where to start?

I’d start first with setting up a Chef Server. The Chef server will provide you with a few things, including a web application that can be used to see all your cookbooks, roles, and nodes at a glance.
Once that’s done, setup a chef repository, that would basically contain the code of your cookbooks. Then, add a few cookbooks, define a first role and assign it some recipes, as well as attributes. Writing cookbooks shouldn’t be too hard, but you should first try to see if someone has written one. Check opscode repository and all the associated branches. With a little luck, you won’t have to write anything but define attributes.
When you have a few cookbooks, clone that chef repo on your chef server and run rake deploy anytime you want to update the configuration.
Then, create a bootstrap script, like this one, which installs the bricks for a chef client on your node.
Finally, start a new node on your favorite cloud provider (Slicehost:“http://slicehost.com”, Linode, EC2), copy the boostrap (scp should make it) script and run it.

The great thing with this “spec” approach is that you can (and should) kill instances all the time and re-configure clean ones in minutes.

Hosting in the cloud really means that you don’t know (or at least shouldn’t care) about the nodes on which your app is running.

I’ve heard people like 37signals, EngineYard or even Twitter are using Chef to scale and deploy their architectures, that can only be a good sign ;)

Again, this was a very light intro, but, when we started to play with Chef, we found that resources (and feedback) were missing.

Superfeedr Java Wrapper

2009-07-28T00:00:00-07:00

This seems to be the wrapper week! Our friend Thomas just pushed a first version of a Java wrapper to our API (his branch). I’d like to take this as an occasion to explain how we see these wrappers.

We know our limits and we know we can’t actively support wrappers in all languages. Like everybody, we have our favorite languages and platform, and there is little chance that we ever get a deep enough knownledge in all that languages that you guys use to offer great services. So, for us, the limit of what we can provide and support is our API (both XMPP and PubSubHubbub).

However, we’re not blind either and we know how these parsers are important for everybody to get started and integrate Superfeedr into their existing apps, so we take great care of them and try to help people use them or fix problems. Aneil helped us a lot on the Python lib, Nathan, Yann or Thomas did an awesome job with the PHP, Perl and Java libs; all of which we would have been unable to make.

These libs are not ours, they wait for your input_, y_our documentation, your testing, as well as your suggestions for new features. We will just help “gathering” them and keeping track by “connecting” the people who made them with the people who use them.

Again, thank you Nathan, Yann, Thomas, Aneil!

PS: anybody for the C# one? ;) (find them all here)

A Perl Wrapper

2009-07-26T00:00:00-07:00

Thanks to our friend Yann, we now have a Perl wrapper for the Superfeedr API (here is yann’s repo if you want faster updates).

What I like with Yann’s wrapper is that it has a callback that users can implement to return the new feeds to subscribe to in a asynchronous fashion.

So, for example, when you need to add a feed to superfeedr, you can save it in your database, and then, implement the callback so that it returns that feed. Any feed that will be returned will be subscribed to. One of the great advantages is that it shouldn’t affect your service performance, since the subscription is then asynchronous.

Yann will soon put that wrapper on CPAN (not yet there, though), but as with any of our other wrappers, you can of course fork it and improve it (or add doc, tests, examples) ;)

PS: You need to use his version of AnyEvent::XMPP to fix a weird defect in the way this lib handles xmpp-urls.

Thank you Yann ! We owe you dinner!

Make Our Logo!

2009-07-25T00:00:00-07:00

A lot of our friends complained about the fact that we didn’t have a logo. That’s a valid concern, but I really didn’t have the skills (even far from it) that it takes. So what I basically asked a very talented friend to make a concept. He did a great job and you can see it right below, with several different colors.

I am releasing it here because I know some of you might have great ideas to actually make it even nicer, and well, I’d love to be able to have feedback and improvements from our actual users.

Here a few words that represent what the logo should convey : superhero, RSS, speed, cloud, reliability, strength, seriousness… Do you see others?

Here are 3 versions of the same logo : AI, EPS, PDF and you can see on our twitter account my own version! What do you think? You can download them and remix them as much as you want!

The Console

2009-07-24T00:00:00-07:00

It is quite hard to debug notification based systems, because developers usually have to wait for a notification to happen. It’s been a common request from several of our users over the past month and we’re happy to say that we released today a solution to this problem : the console.

Basically, when logged in, you should see ‘Your feeds’ in the top bar. Click there, and then, at the bottom of the page, click on “Console”.

The console is a tool that will enable you to see notifications directly pushed to your browser, exactly as they would be pushed to your XMPP client (technically, the console IS an XMPP client).

To see actual notifications happening, we’ve also created a dummy atom feeds. In the console interface, you should see a link that, when clicks, publishes a new entry in this dummy feed. A few seconds (or less) later, you should see the new notification in your console. Of course, that works with any feed, so you can as well add others.

Here a a few tips to make that console even more useful :

The console only works if your using using your default jid : user@superfeedr.com (no webhooks, no ‘private’ jid)
Add http://superfeedr.com/dummy.xml to your feeds (and more feeds)
Change your notification mode (See ‘settings’) to “Each time we pull and parse one of your feeds”
Leave the console open for a few hours : notifications will stack up (most recent at the top)

Kind of Open Source

2009-07-22T00:00:00-07:00

When I tweeted about sharing our Investor Deck yesterday, I received immediately a few IM and email of friends who wanted to make sure I was doing the right decision. They were scared that I would make some private things public. I fully understand their concern, but I really want Superfeedr to be an Open Service, where we are guided by our users more than our wallet. We value their opinion more than ours because in the end, they are going to pay for our service, not us.

Why not release our software as Open Source, then?

Because that wouldn’t work. The idea with Superfeedr is to do something stupid so that nobody else has to do it. If we release our software, then, everybody would run their standalone superfeedr and people will keep doing this inefficient polling that wastes our resources.
Superfeedr will work only with massive economies of scale.

We will eventually release huge parts (and we already started) of our software as open source, as we will probably communicate and ask advices on as much as we can that relates to Superfeedr. If you’d like to give a look at our deck, send us your email as a direct message. Also, follow us on Twitter, because I will (thanks to @jineshmehta) release there the master file of our logo. I hope you can then make it even nicer. (BTW, red, or green?)

You are the brains of our service, giving the ideas, suggesting the directions. We are just the ~~small~~ hands selecting, organizing and implementing ideas.

Reality checked

2009-07-21T00:00:00-07:00

A lot of people have asked us why we built Superfeedr or what is the relationship to Notifixious. Here are a few hints.

A little over a year we launched Notifixious. The idea behind it was to allow any internet user to receive real-time updates from his favorite websites. We would send these updates either by email, SMS or Instant Messaging. For example, a user might subscribe to a blog to receive the last updates, or to a newspaper site. A user could also subscribe to a Craigslist search to be the first to know about a new place to move in, or to Simply Hired to get the last jobs in his field of expertise.

Earlier this year, we did an extensive analysis of our user base and found out 2 patterns.

About 60% of our users would select email to receive their notifications. On average they would subscribe to 1 single website and most of the time it was a blog. These users have clearly been acquired through our widget. We were very disappointed : email is not real-time and the email user-experience is usually bad : spam, phishing… etc. We asked those people why they chose email and the answer was clear : they are scared by information overload and have no idea how to deal with an incoming stream of notifications which is very stressful, because they know they wanted to receive these notifications.
A dozen of our users had, on the other hand, subscribed to more than 500 feeds. Oddly enough they were all using our XMPP api or a jabber account to receive these notifications. This behavior was very interesting compared to the first group. We contacted most of them and understood that they were actually using notifixious as a middleware for their own application or service. In a nutshell, they were using Notifixious as an on-demand feed parsing service. That was a very insightful discovery since some of them even said they would pay for our service. The idea to extract Superfeedr out of Notifixious was born.

We still haven’t identified how to deal with the first group, except that real-time services will need to find a way to avoid that information overload (or at least the sensation of it). However, by observing and understanding the behavior of a ~~very~~ small set of users, we’ve been able to identify some of the value of our service.

We even believe that this very small subset of our user base carried the biggest of the value our service provided. By continuously looping the feedback with this small user base, we think we can provide a great service!

Federating PubSubHubbub

2009-07-12T00:00:00-07:00

PubSubHubbub has been the RealTime web celebrity in the past few days. Google announced their support for Feedburner and Blogger, and that’s great to see the interest growing.

We decided a few weeks ago that PubSubHubbub would be our HTTP API and it’s been a great choice for us as many developers have started to work on language specific libraries to interact with hubs.

Today, we’re announcing that not only we’re a hub, but we’re also a subscriber. In other terms, it means that whenever a feed indicates a hub, Superfeedr will subscribe to this hub to get updates from it in real-time. That is a great addition to our set of helpers and it will hopefully help us improve yet again our detection time of new entries.

At a broader level, it is interesting to compare PubSubHubbub with previous initiatives, and, as far as I know, PubSubHubbub is the first one to actually enable easily this “federation” structure (similar to what XMPP offers), where subscribers can select their favorite hubs and hubs can communicate together to provide updates to all their users.

It makes the PubSubHubbub “network” even more decentralized and scalable/fault-resistant. For those into graph theory, you will immediately see the benefit of such an architecture : virtually ubiquitous new data publication, as long as there is a (not too long) path from any node to any other node!

Customer Care Lesson

2009-07-10T00:00:00-07:00

Yesterday, the Carsonified team relaunched their website and it’s looking great. The Mission page should be a lesson for everybody:

At Carsonified we believe in treating other people like we want to be treated: with friendliness, patience and respect. We believe that the web industry is one of the most exciting places in the world, and that it’s an honor for us to connect, encourage and train that community through our events. We believe in being more creative than we need to be and having fun while we’re at it.

There is no such thing as Real Time Search

2009-07-09T00:00:00-07:00

Real-time search seems to be the most common tagline these days. Real-time is the new hotness (we agree!) and Google almost made ‘Revenue’ and ‘Search’ synonyms. I guess it was tempting to mix both ;)

When I am doing a “search” on Google or Bing (try it!), I technically don’t know what I will find. To be more precise, I don’t know where I will find the information, and I even don’t know if what I am looking for exists.

Twitter Search (ex-summize) is clearly the best known example of Real-time Search and Collecta is a good “cross-services” examples. I wouldn’t call any of them search engines, but rather “filters”.

In both cases, I do searches about things that are happening (Jackson’s death, Palin’s resignation…), which proves they exist. I also tend to know what I’ll find : images, links to more in-depth articles… etc. I am just trying to “filter” the stream of information, not searching for an answer.

In the end, both traditional search-engines and real-time filters have to deal with relevance, but I think the user’s expectations are very different in both cases. What do you think?

Speeding Up RSS : what for?

2009-07-09T00:00:00-07:00

Erick Schonfeld published the blog post of the ~~day~~ week. Despite the fact that he should have talked about Superfeedr, I think this is an awesome article. I am really looking forward to tomorrow’s panel and see what “Brett”, Brad, Freddy and Andy have to talk about.

The comments, though are very interesting and they show what we’ve observed with "Notifixious"http://notifixio.us over the last year : Real-time notification (or faster RSS) scares people a lot. There is at least 4 or 5 comments saying that RSS is fast enough for them. The problem here, is that there is a misconception :* making RSS faster is not for individuals, but for the web itself*.

PubSubHubbub has the potential to bring Push to the RSS and Atom technologies. So not only it will be faster, but it will be instantaneous, and when Feed updates will be instantaneous, nobody will pull anymore (not even us) and the web will be better.

Better? Publishers will not have to handle massive loads anymore (like Twitter had to), and Subscribers (services) will not have to poll over and over to get updates.

The future of Web apps?

2009-07-05T00:00:00-07:00

A lot of people have shown interest in Aristochat and the Superfeedr chat last week. However, I haven’t talked about what is – according to me – the greatest thing about it : it’s hosted on Github. Not only the source code, but the app itself, and since it’s a basic HTML file with a little bit of Javascript, it could have very been hosted on EC2 as well, or on any type server that can server HTML file.

This blog is also a great example : technically, it’s an app, and yet, it’s made of static HTML files, hosted on Github as well. I think these 2 examples can give us a “hint” on what Web apps will look like in a not so distant future :

A bunch of static HTTP files with CSS, served by clusters
A lot of Javascript inside these pages to enhance the user experience with effects, drag and drops… etc
Some AJAX to send and receive data from one (or more) application servers, through APIS (JSON, or XML)

The great thing about that is that web services (and developers) can focus on a single “job”, while designers, UX engineers can ‘pick’ up whatever tool they want to integrate into an app and make it richer.

With this approach, you can really decouple more and more the data and the processing from the views. The APIs can very well be consumed by a webservice, but also by, say, an iPhone App, an Adobe Air App, or even external web services and we sites.

This should sound familiar, doesn’t it? It’s very close from the MVC model that we all learnt to create desktop apps.

Aristochat

2009-07-03T00:00:00-07:00

It all started when we used Slicehost’s chat-room for their support : not only it’s awesome to have 24/7 support, but it’s also great to have a short-message interface that allows real-time discussion and interaction.

Unfortunately, Campfire is “proprietary” software and it doesn’t seem to be XMPP based. XMPP is known as THE instant messaging protocol, and it has the fairly well defined and complete Multi User Chat Extension .

Also, XMPP has Bosh, which allows transportation of XMPP stanzas over HTTP. From that, there is no reason to think we couldn’t do a Web Based Multi User Chat, with just 1 HTML page, and 1 Javascript and that could be integrated in any web framework in just a few seconds : meet Aristochat.

The great thing about Aristochat is that it’s just a MUC client, which means that you can connect (and manage) to the room as well from your favorite XMPP client (like Psi Gajim or Adium).

Check the Superfeer chat to see an example. And, yes, Aristochat is OSS, so you can fork, fetch and push it back with more features!

User Test Driven Development

2009-06-30T00:00:00-07:00

We are strong believer of Test Driven Development. It’s a very powerful approach that allows you to write tests before actually writing code. (These tests are actually specifications). That is how we created our parsers : we selected random feeds on the web, and decided what should be the mapping between their data and our standard Atom schema. Then, we wrote the code to actually perform the mapping.

Unfortunately, the web isn’t standard (that’s actually one of the problems we tackle!), so there are probably feeds out there for which the mapping that we’ve identified isn’t what you would expect us to do. You can help us improve our mapping to fit your needs.

In our test suites we have YAML files representing these mappings and their corresponding XML files. Then, we run it by comparing the parsed document to the YAML mappings.

Here is and XML feed :

Here is a example of mapping :

If you have identified a feed that you think we’re not parsing correctly, please, send us a this feed.xml file, along with the corresponding specification yaml file. We will then fix our code so that we extract the right info from your file.

You can use this template for the mapping file.

Building Realtime : turning feeds into streams

2009-06-28T00:00:00-07:00

Technically, Real-Time Computing implies that there is a time constraint in a system, it’s different from instantaneity. A real-time service is a service that can guarantee that it would do something in a given amount of time. Superfeedr is real-time, since we can guarantee that we will notify new entries in a feed to our users within a 15 minutes time frame.

This is only achievable if we “push” the information toward our users. We couldn’t guarantee delivery time if we asked them to “pull” from us¹.

This is a keyword here : in a real-time web, information is pushed from a place to another one. Facebook pushes my friend updates to my homepage, Twitter search pushes new result to the result page, Friendfeed pushes the conversation to a topic’s page… etc.

Unfortunately, as the web is right now, the information is almost never pushed to anybody. Look at this blog. I will publish this blog post and wait for you, for Google, for search engines to actually pull its content. Similarly, these services, once they have the information will probably not push it to anyone.

In a perfect world, not only I would push this new content to you readers, to Google, to Delicious, to Digg, to Twitter, but also these services would push an update to their users and other services to notify them that received something from superfeedr… and so on.

Our mission at Superfeedr is to take all these ‘static’ bits of information that are feeds, and push them to your services so that you can as well push them (or derived information) to other services… creating a stream.

We’re turning feeds into stream.

¹ On top of that, pull is stupid .

A better Schema

2009-06-25T00:00:00-07:00

One of the great features of Superfeedr is that it sends a standard format for feed entries. Whether the initial feed format was any flavor of Atom, or RSS, we would map this into a standard Atom feed. Thanks to that it is easier for you to integrate this data into your apps, since you will gain consistency in the format.

However, to achieve that, we have to make some decisions and scraps information from the initial feed. Mapping a title is easy, but that’s about the only obvious data. We actually had to find the smallest common set of attribute that a feed has.
Up to now, entries that we sent had this :

ID
URL
Title
Summary
Published date

We’ve been working to complete that over the past few weeks, and here is what you will soon get, on top of what you already have:

Content (if available, if not, you’ll get a summary)
Categories (or tags)
Authors informations : email, name and uri
Links : not only you will get the entry’s url, but you will also get any other link that the entry may contain : enclosure, comments… etc.

The new schema will be published soon, but first, we want to make sure that there is no other element that you would want to see added to it. What would you add in there?

RSS or Atom, not both!

2009-06-23T00:00:00-07:00

RSS and ATOM have very similar (if not identical) purposes. I don’t know any feed consumer that consumes only one type, yet, a lot of services are still providing both.

Publishing both is useless, because it’s likely that services will end up polling both of them, to identify new content. It is very hard for them to be 100% sure that they actually contain (and will always contain) the same information. You know that polling your blog or service is stupid… don’t make twice as stupid.

Yet, if you publish both ATOM and RSS, make our lives easier in two steps :

Remove the extra <link href="...">, by keeping just Atom or RSS

Add a 301 redirection from one to another, so that you’re not messing with your existing pollers (as they should probably follow redirections and be able to parse either format) : with Apache it’s as easy as: RedirectMatch 301 /statuses/user_timeline/(.*)\.rss http://twitter.com/statuses/user_timeline/$1.atom

No more excuse for not doing it! Please note that I am not discussing whether you should keep ATOM or RSS. I’d go for Atom, but that’s a not the case for everybody.

We're doing something stupid so that you don't have to

2009-06-19T00:00:00-07:00

It’s a fact, polling sucks, and it sucks, because it’s stupid. This is what kellan and rabble showed last year at OSCON :

On July 21, 2008, Friendfeed hit Flickr 2.9 million times to get the latest photos of 45,754 users, of which 6,721 visited Flickr in that 24-hour period, and could have potentially uploaded a photo.

And this is just friendfeed. How many services is there around here that are polling Flickr, Twitter, Delicious, Youtube, or even this blog? Thousands (except maybe for this blog).

Push is the solution. Except that it only works if each service out there pushes, and if each consumer actually accepts to receive pushed content. That makes it pretty hypothetical, right?

We (pollers) should collaborate. Collaboration means that, even if we can all poll, there is probably no need for that. We could probably all grow tomatoes, but it makes more sense to buy them from someone who can grow them cheaper and more effectively than us.

This is what Superfeedr is all about: ~~growing tomatoes~~ doing something that you could do, but doing it cheaper, more efficiently and, more importantly, by sharing the results with you.

Yes, polling is still stupid, but we’re doing it for you and everybody who cares about the same feeds.

PubSubHubbub

2009-06-18T00:00:00-07:00

We’re proud to announce that Superfeedr is one of the very first services to actually support the infamous PubSubHubbub protocol. If you think the name is ugly, check the CSS style of the spec.

This presentation should give you enough details about PubSubhubbub.

What does that mean for you superfeedr users? It means that you can get the ATOM payload of the feeds we parsed pushed to your endpoint url. Check our implementation notes to learn more.

Ha, I almost forgot, you can find our hub there http://superfeedr.com/hubbub.

Hello World

2009-06-17T00:00:00-07:00

Welcome to the Superfeedr’s blog. I am not sure what you expect from here. You know what a geek does when he’s embarrassed does, right? He gets into technical details.

This blog has been built with the great piece of code that Jekyll is. It is hosted on Github, and I used TextMate Version 1.5.8 (1498) to write it. I also used Git to push it on our Github repository. TextMate runs on my MacBookPro 2.53 Ghz Intel Core 2 Duo, with 4 GB of DD3 RAM. It’s about 85° F in the room in which I am working right now.

Welcome!

Superfeedr Blog : Real-time cloudy thoughts from a super-hero

Chasing Real-Time Raindrops in an Ocean of Content

Ping me I'm famous

Building a Ping Cache.

Fetching the feed.

A Hub for Twitter

Would you use a Twitter PubSubHubbub hub if it was available? Yes 45 96% No 2 4%

Do you currently use the Twitter Streaming API? Yes 21 45% No 26 55%

Do you think a hub would increase your Twitter data consumption? Yes 45 96% No 2 4%

Have you already implemented PubSubHubbub? Yes 30 64% No 17 36%

Do You Prefer Buffet or Sit-Down?

On Outsourcing

UrbanAirship

What’s your product/service?

How do you use Superfeedr for?

Why do you use Superfeedr? (vs building your own!)

What’s your stack? Language? Superfeedr API you use?

What would you change if you were in charge of Superfeedr for 1 day?

Redis at Superfeedr

How it is/was

Requirements

Implementation

Tip

Conclusion

Google's Buzz is PubSubHubbub in, and out

Huh, and the rest of us?

PubSubHubbub : security

Gawker Media and ReadWriteWeb have hubs

10 Feed Publishing Best Practices

State of Real-Time feeds

Why realtime matters for publishers

Now

In realtime

A hub for Dotspots

Compliance for Superfeedr's Hubs

Who Implemented PubSubHubbub

Publisher Callback Extension to PubSubHubbub

XML-RPC Ping to PubSubHubbub

Scaling Superfeedr

Feedoor and Superfeedr

Who Implemented PubSubHubbub

Distributed and Federated Social Networks

Be my guest

Feediscovery

Example:

How we built Analytics

Collecting

Storage

Querying

Displaying

Tumblr has a hub

Feedburner stats for real-time web

LeWeb's Official Bloggers in RealTime

How-to

Result

Results of the contest

#3 Superfeedrfall

#1 (ex-aequo) Rss.im’s stream

#1 (ex-aequo) AppNotification’s iPhone app

The Real-time Web and it's future

On the report itself

Real-time : the peer-to-peer architecture.

State of Real-Time feeds

Thanks

Added Title to Feed Info

Seed Round

Win a ticket for LeWeb!

SUP over PubSubHubbub and XMPP

The Non-Decision makers

They don’t want to make decisions

Why no is better than maybe

A few tips

Posterous supports PubSubHubbub

Sunday Package

Async Subscription supported with PubSubHubbub

A Facebook fan page

Another Ruby wrapper

Notifixlite

Funny Feeds

Never Ending HTTP responses

Would you use a Twitter PubSubHubbub hub if it was available?
Yes 45 96%
No 2 4%

Do you currently use the Twitter Streaming API?
Yes 21 45%
No 26 55%

Do you think a hub would increase your Twitter data consumption?
Yes 45 96%
No 2 4%

Have you already implemented PubSubHubbub?
Yes 30 64%
No 17 36%