When I had heard about twitter initially the question that had popped in my mind was What the hell is twitter?, at this point the question I have in my mind is What the hell is twitter doing?, though after two years the question has not changed drastically but the meaning definitely has.

There have been several downtimes with twitter of late and we even showed you the funny way you can take in the downtimes, but this problem is serious and nevertheless needs a bit of thinking by the folks at twitter. They have to sit down and think how they can grow horizontally by supporting everything that they have right now, before some other service bulldozes them to death.

There has been a exponential growth of people who have started using twitter in the past 6 months and the infrastructure twitter had in place has simply collapsed. Can we blame that on bad specifications they had in the first place? Absolutely no, twitter was doing fine just 6 months ago, the sudden surge of usage has almost led them to their knees.

In my opinion I blame twitter team for not sitting down and thinking about scalability, adding new hardware to a system definitely does not solve scalability problems as many of you perceive, if the underlying software needs upgrading no matter how many servers you keep on adding, you are bound to fail.

A example where hardware can only scale as good as the software is, can be accomplished by having a PC with a Quad 2 Core processor and 4 GB ram running on a Windows 98 operating system, of course Windows 98 would never be able to utilize the hardware capabilities, because the underlying software is incapable of doing it. You would have to upgrade your operating system to XP 64 bit or Windows Vista before you can ensure full utilization of your hardware.

There have been several instances in my IT career of 8 years, where people have simply changed the underlying software and database to solve scalability issues. So Ruby on Rails had to go a long time ago and be replaced by a more scalable software like Java or Python which would do good in the long term. This exact question about scalability with ROR had been talked about in a interview with a twitter developer back in March 2007.

On the other hand I do not have any issues with using MySQL and PostgreSQL, and have been using for several of the applications I create, but using it has led me to a conclusion that it’s simply not scalable in a larger environment, replacing the database with something like Oracle will do a world of good.

So drawing a end to my ranting, I have to say that twitter needs to get its underlying software scalable before allowing any more new developers to tap into their API, just lock the API down and get working on your own software. This may piss off several developers but given a month or two, once twitter makes the underlying changes everyone will be happy.

Do you have your own opinions about this, let me know through your comments.