It’s hard to talk about numbers, because they’re often subject to interpretation. I was crunching a few of them earlier today for some presentation that I’m working on and I found out that Superfeedr is currently pushing away about 237 new entries per second.
On a daily basis, this is 237 * 3 600 * 24 = 20,476,800.
I would argue that a tweet is the rough equivalent of an Atom entry (much shorter usually.) Back in February, Twitter was pushing about 50 million tweets daily. Yes, we’re at 40% of Twitter’s scale in number of messages pushed.
Obviously, we just push the data and don’t have to handle incoming requests, like twitter does. But we could we could very well be pushing all this data to S3 and let Amazon deal with the scaling issues of having client poll them :) At the same time, we do a hell of a lot of polling, which twitter doesn’t.
I’m not sure we could double up with the current number of servers, but based on the fact that we have less than 0.8‰ (that’s about 8 for 10k feeds) of “late” feeds in our scheduling algorithm, I will say our capacity utilization is low.
Our architecture for doing this is built at 90% on XMPP. It’s basically a network of 120 XMPP clients who work together (on 35 512MB slices from Slicehost). Add 2 XMPP servers, 2 MySQL servers, 10 Redis data stores and you’ve got pretty much the whole Superfeedr task force.
We want to push all these limits even further! Send us your data!
Comments